Fatigue measurement method, apparatus, and computer-readable medium

Patent metadata

Field	Value
Publication number	US-12324666-B2
Application number	US-202217678951-A
Country	US
Kind code	B2
Filing date	Feb 23, 2022
Priority date	Jun 2, 2021
Publication date	Jun 10, 2025
Grant date	Jun 10, 2025

Abstract

A fatigue measurement method includes obtaining a face video of a target object, processing the facial video to obtain a plurality of quasi-heart rates and a plurality of quasi-vibration frequencies, selecting a target heart rate from the plurality of quasi-heart rates and a target vibration frequency from the plurality of quasi-vibration frequencies, detecting the target heart rate and the target vibration frequency, and determining, according to the detection result, that the target object is in a fatigue state. The quasi-heart is obtained through one or more of a region of interest (ROI) between eyebrows and an ROI of chin, and the quasi-vibration frequency obtained through one or more of an ROI of eyes and an ROI of mouth.

First claim

What is claimed is:

1. A fatigue measurement method, comprising:
obtaining a face video of a target object;
processing the facial video to obtain a plurality of quasi-heart rates and a plurality of quasi-vibration frequencies, wherein the quasi-heart is obtained through one or more of a region of interest (ROI) between eyebrows and an ROI of chin, and the quasi-vibration frequency is obtained through one or more of an ROI of eyes and an ROI of mouth;
selecting a target heart rate from the plurality of quasi-heart rates and a target vibration frequency from the plurality of quasi-vibration frequencies; and
detecting the target heart rate and the target vibration frequency, and determining, according to the detection result, that the target object is in a fatigue state,
wherein processing the face video to obtain the plurality of quasi-heart rates and the plurality of quasi-vibration frequencies, comprising:
dividing the face video into a plurality of window videos according to a time window of a specific step size; and
performing, for each window video of the plurality of window videos, extraction processing on the each window video to obtain an eye feature signal, a mouth feature signal, an eyebrow feature signal, and a chin feature signal, fusing the eye feature signal, the mouth feature signal, the eyebrow feature signal, and the chin feature signal to obtain a fused feature signal, and converting the fused feature signal to obtain a quasi-heart rate and a quasi-vibration frequency.

2. The method according to claim 1 , wherein performing the extraction processing on the window video to obtain an eye feature signal, a mouth feature signal, an eyebrow feature signal, and a chin feature signal, comprising:
using the model to predict the window video to generate face key points, the face key points including contour key points and specific part key points;
extracting eye key points, mouth key points, the ROI between the eyebrows, and ROI of the chin respectively from the face key points;
performing dimension reduction processing on the eye key point and the mouth key point respectively to obtain an eye feature signal and a mouth feature signal;
performing feature extraction on the ROI between the eyebrows and the ROI of the chin, respectively to obtain an eyebrow color signal and a chin color signal; and
performing noise reduction processing on the eyebrow color signal and the chin color signal respectively to obtain an eyebrow feature signal and a chin feature signal.

3. The method according to claim 1 , wherein performing the conversion processing on the fusion feature signal to obtain the quasi-heart rate and the quasi-vibration frequency, comprising:
performing noise reduction processing on the fusion feature signal to obtain a smooth feature signal;
performing blind source separation on the smooth feature signal, and converting signal components of the smooth feature signal after the blind source separation into frequencies for obtaining a plurality of frequencies;
selecting, based on a body heart rate range, a pre-heart rate from the plurality of frequencies, and averaging the selected pre-heart rate to obtain a quasi-heart rate; and
selecting, based on the eye vibration frequency range and mouth vibration frequency range, a pre-vibration frequency from the plurality of frequencies, and averaging the selected pre-vibration frequencies to obtain a quasi-vibration frequency.

4. The method according to claim 1 , wherein the selecting the target heart rate from the plurality of quasi-heart rates and selecting the target vibration frequency from the plurality of quasi-vibration frequencies comprises:
determining a median of the plurality of quasi-heart rates and a mean value of the plurality of quasi-vibration frequencies;
determining, for each of the plurality of quasi-heart rate, a difference between the each quasi-heart rate and a corresponding median, and determining whether the difference is greater than a first preset heart rate threshold, and in response to the difference being greater than the first preset heart rate threshold, removing the corresponding quasi-heart rate from the plurality of quasi-heart rates to obtain the target heart rate; and
determining, for each of the quasi-vibration frequencies, a difference between the quasi-vibration frequency and a corresponding median, and determining whether the difference is greater than a first preset vibration threshold, and in response to the difference being greater than the first preset vibration threshold, removing the corresponding quasi-vibration frequency from the plurality of quasi-vibration frequencies to obtain the target vibration frequency.

5. The method according to claim 1 , wherein the detecting the target heart rate and the target vibration frequency, and determining that the target object is in a fatigue state according to the detection result comprises:
obtaining a normal heart rate and a normal vibration frequency of the target object when the target object is in a non-fatigue state;
determining a difference between the normal heart rate and the target heart rate and a difference between the normal vibration frequency and the target vibration frequency respectively;
determining, within preset measurement times, in response the difference corresponding to the target heart rate measured each time being greater than a second preset heart rate threshold and the difference corresponding to the target vibration frequency measured each time being greater than a second preset vibration threshold, that the target object is in a mild fatigue state.

6. The method according to claim 1 , wherein the detecting the target heart rate and the target vibration frequency, and determining that the target object is in a fatigue state according to the detection result comprises:
obtaining a normal heart rate and a normal vibration frequency of the target object when the target object in a non-fatigue state;
determining a difference between the normal heart rate and the target heart rate and a difference between the normal vibration frequency and the target vibration frequency respectively;
determining, within preset measurement times, in response to the difference corresponding to the target heart rate measured each time being greater than a third preset heart rate threshold and a difference corresponding to each measured target vibration frequency being greater than a third preset vibration threshold, that the target object is in a microsleep state.

7. A fatigue measurement apparatus, comprising:
an acquisition module configured to obtain a face video of a target object;
a processing module configured to process the face video to obtain a plurality of quasi-heart rates and a plurality of quasi-vibration frequencies, wherein the quasi-heart rate is obtained through one or more of a region of interest (ROI) between eyebrows and an ROI of chin, and wherein the quasi-vibration frequencies is obtained through one or more of an ROI of eyes and an ROI of mouth;
a selection module configured to select a target heart rate from the plurality of quasi-heart rates and a target vibration frequency from the plurality of quasi-vibration frequencies; and
a detection module configured to detect the target heart rate and the target vibration frequency, and determine that the target object is in a fatigue state according to the detection result,
wherein the processing module comprises:
a dividing unit configured to divide the face video into a plurality of window videos according to a time window of a specific step size; and
a processing unit configured to, for each of the window videos, perform extraction processing on the each window videos to obtain an eye feature signal, a mouth feature signal, an eyebrow

Assignees

• Lenovo Beijing Ltd

Inventors

• Han Xing
• Liu Wenrui

Classifications

• G06F18/253

  of extracted features · CPC title

• A61B5/7203

  for noise prevention, reduction or removal · CPC title

• G06F2218/08

  Feature extraction · CPC title

• G06F2218/04

  Denoising · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title