Face-hand correlation degree detection method and apparatus, device and storage medium

Patent metadata

Field	Value
Publication number	US-11847810-B2
Application number	US-202117489941-A
Country	US
Kind code	B2
Filing date	Sep 30, 2021
Priority date	Sep 16, 2021
Publication date	Dec 19, 2023
Grant date	Dec 19, 2023

Abstract

Provided are a face-hand correlation degree detection method and apparatus, a device, and a storage medium. The method includes that: an image to be detected is acquired; a face feature set and a hand feature set of the image to be detected are determined on the basis of a result obtained by performing face and hand detection on the image to be detected; a first interaction feature of a target face is determined on the basis of a face feature of the target face and the hand feature set; a second interaction feature of a target hand is determined on the basis of a hand feature of the target hand and the face feature set; and a correlation between the target face and the target hand is determined on the basis of the first interaction feature and the second interaction feature.

First claim

What is claimed is:

1. A face-hand correlation degree detection method, comprising:
acquiring an image to be detected;
determining a face feature set and a hand feature set of the image to be detected on the basis of a result obtained by performing face and hand detection on the image to be detected, wherein each face feature in the face feature set corresponds to one face in a picture of the image to be detected, and each hand feature in the hand feature set corresponds to one hand in the picture of the image to be detected;
determining a first interaction feature of a target face on the basis of a face feature of the target face and the hand feature set, wherein the target face is any face in the picture of the image to be detected, and wherein the first interaction feature is obtained by fusing the face feature of the target face with hand features, belonging to the hand feature set, of all hands in the picture of the image to be detected;
determining a second interaction feature of a target hand on the basis of a hand feature of the target hand and the face feature set, wherein the target hand is any hand in the picture of the image to be detected, and wherein the second interaction feature is obtained by fusing the hand feature of the target hand with face features, belonging to the face feature set, of all faces in the picture of the image to be detected; and
determining a correlation between the target face and the target hand on the basis of the first interaction feature and the second interaction feature.

2. The method of claim 1 , wherein the determining a face feature set and a hand feature set of the image to be detected on the basis of a result obtained by performing face and hand detection on the image to be detected comprises:
determining a face detection box of each face and a hand detection box of each hand in the picture of the image to be detected on the basis of the result obtained by performing the face and hand detection on the image to be detected;
extracting a feature of the each face based on the face detection box of the each face to obtain the face feature set; and
extracting a feature of the each hand based on the hand detection box of the each hand to obtain the hand feature set.

3. The method of claim 1 , wherein the determining a first interaction feature of a target face on the basis of a face feature of the target face and the hand feature set comprises:
constructing a first undirected graph on the basis of the target face and each hand in the picture, wherein the first undirected graph includes a first node corresponding to the target face, a second node in one-to-one correspondence with the each hand, and a first side in one-to-one correspondence with the second node, each first side is configured to connect the first node and one of second nodes;
determining a first correlation degree between a hand feature of the hand corresponding to the second node connected to the each first side and the face feature of the target face in the first undirected graph; and
determining the first interaction feature on the basis of the face feature of the target face, the hand feature of the each hand in the picture, and the corresponding first correlation degree.

4. The method of claim 3 , wherein the determining a first correlation degree between a hand feature of the hand corresponding to the second node connected to the each first side and the face feature of the target face in the first undirected graph comprises:
determining a first confidence degree that the hand corresponding to the second node connected to the each first side and the target face belong to a same body on the basis of the hand feature of the hand corresponding to the second node connected to the each first side and the face feature of the target face; and
normalizing the first confidence degree that the hand corresponding to the second node connected to the each first side and the target face in the first undirected graph belong to the same body, to obtain the first correlation degree between the hand feature of the hand corresponding to the second node connected to the each first side and the face feature of the target face.

5. The method of claim 4 , wherein the determining a second interaction feature of a target hand on the basis of a hand feature of the target hand and the face feature set comprises:
constructing a second undirected graph on the basis of the target hand and each face in the picture, wherein the second undirected graph includes a third node corresponding to the target hand, a fourth node in one-to-one correspondence with the each face, and a second side in one-to-one correspondence with the fourth node, each second side is configured to connect the third node and one of fourth nodes;
determining a second correlation degree between a face feature of the face corresponding to the fourth node connected to the each second side and the hand feature of the target hand in the second undirected graph; and
determining the second interaction feature on the basis of the hand feature of the target hand, the face feature of the each face in the picture, and the corresponding second correlation degree.

6. The method of claim 5 , wherein the determining a second correlation degree between a face feature of the face corresponding to the fourth node connected to the each second side and the hand feature of the target hand in the second undirected graph comprises:
determining a second confidence degree that the face corresponding to the fourth node connected to the each second side and the target hand belong to a same body on the basis of the face feature of the face corresponding to the fourth node connected to the each second side and the hand feature of the target hand; and
normalizing the second confidence degree that the face corresponding to the fourth node connected to the each second side and the target hand in the second undirected graph belong to the same body, to obtain the second correlation degree between the face feature of the face corresponding to the fourth node connected to the each second side and the hand feature of the target hand.

7. The method of claim 6 , wherein the determining a correlation between the target face and the target hand on the basis of the first interaction feature and the second interaction feature comprises:
determining a third confidence degree that the target face and the target hand belong to a same body on the basis of the first interaction feature and the second interaction feature; and
determining the third confidence degree as a correlation degree between the target face and the target hand.

8. The method of claim 5 , wherein the determining the second interaction feature on the basis of the hand feature of the target hand, the face feature of the each face in the picture, and the corresponding second correlation degree comprises:
adjusting the face feature of the each face on the basis of the second correlation degree corresponding to the each face, to obtain an adjusted feature of the each face; and
fusing the adjusted feature of the each face in the picture and the hand feature of the target hand to obtain the second interaction feature.

9. The method of claim 3 , wherein the determining the first interaction feature on the basis of the face feature of the target face, the hand feature of the each hand in the picture, and the corresponding first correlation degree comprises:
adjusting the hand feature of the each hand on the basis of the first correlation degree corresponding to the each hand, to obtain an adjusted feature of the each hand; and
fusing the adjusted feature of the each hand in the picture and the face feature of the target face to obtain the first interact

Assignees

• Sensetime Int Pte Ltd

Inventors

• Liu Chunya
• Zhang Xuesen
• Wang Bairun
• Chen Jinghuan

Classifications

• G06V10/25 Primary

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06F18/253

  of extracted features · CPC title

• G06N3/08

  Learning methods · CPC title

• G06V40/107

  Static hand or arm · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title