Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions

Patent metadata

Field	Value
Publication number	US-10335045-B2
Application number	US-201715631346-A
Country	US
Kind code	B2
Filing date	Jun 23, 2017
Priority date	Jun 24, 2016
Publication date	Jul 2, 2019
Grant date	Jul 2, 2019

Abstract

Recent studies in computer vision have shown that, while practically invisible to a human observer, skin color changes due to blood flow can be captured on face videos and, surprisingly, be used to estimate the heart rate (HR). While considerable progress has been made in the last few years, still many issues remain open. In particular, state-of-the-art approaches are not robust enough to operate in natural conditions (e.g. in case of spontaneous movements, facial expressions, or illumination changes). Opposite to previous approaches that estimate the HR by processing all the skin pixels inside a fixed region of interest, we introduce a strategy to dynamically select face regions useful for robust HR estimation. The present approach, inspired by recent advances on matrix completion theory, allows us to predict the HR while simultaneously discover the best regions of the face to be used for estimation. Thorough experimental evaluation conducted on public benchmarks suggests that the proposed approach significantly outperforms state-of-the-art HR estimation methods in naturalistic conditions.

First claim

What is claimed is:

1. A method of determining heart rate through observation of a human face, comprising:
acquiring with at least one automated camera, a time series of images of a human face, wherein the time series of images are subject to variations between respective images of the time series in illumination and facial movements;
adaptively selecting, with the at least one automated processor, a subset of the regions of interest of respective images of the time series of images of the human face, that exhibit a more statistically reliable heart-rate-determined variation than a non-selected subset of regions of the respective images of the human face;
based at least on the adaptively selected subset of regions of interest of the respective images of the time series of the human face that exhibit the reliable heart-rate-determined variation, determining a heart rate, and updating the adaptively selected subset of the regions of interest that exhibit the reliable heart-rate-determined variation; and
outputting a signal corresponding to the determined heart rate.

2. The method according to claim 1 , wherein the regions of interest are selected according to at least matrix completion theory.

3. The method according to claim 1 , wherein the heart rate is determined based on at least matrix completion theory.

4. The method according to claim 1 , wherein the selected subset is selected dependent on at least a noise parameter of respective features of the time series of images.

5. The method according to claim 1 , wherein the selected subset is selected dependent on at least a movement of the human face represented in the time series of images.

6. The method according to claim 1 , wherein the selected subset is selected dependent on at least changes represented in the time series of images which represent human facial expressions.

7. The method according to claim 1 , further comprising tracking the face in the time series of images to follow rigid head movements.

8. The method according to claim 1 , further comprising detecting chrominance features from the time series of images comprising video images, and assessing the heart rate-determined variation based on the detected chrominance features.

9. The method according to claim 1 , wherein the adaptively selected subset of the regions of interest exhibit the reliable heart-rate-determined variation through an entire period of heart rate estimation.

10. The method according to claim 1 , wherein the reliable heart-rate-determined variation is a variation in chrominance.

11. The method according to claim 1 , wherein the heart rate is determined in a process employing a cardiac cycle responsive filter.

12. The method according to claim 1 , further comprising simultaneously recovering an unknown low-rank matrix and an underlying data mask, corresponding to most statistically reliable heart-rate-determined variation observations of the human face according to a reliability statistic.

13. A method of determining heart rate from video images, comprising:
processing, with the at least one automated processor, a stream of video images of a face from at least one automated camera, to extract a plurality of face regions;
computing chrominance features of the plurality of face regions, with at least one automated processor;
jointly estimating an underlying low-rank feature matrix and a mask of a selected subset of the plurality of face regions which have a higher statistical reliability than a non-selected subset of the plurality of face regions, using a self-adaptive matrix completion algorithm, with the at least one automated processor; and
computing the heart rate from a signal estimate provided by the self-adaptive matrix completion algorithm, with the at least one automated processor.

14. The method according to claim 13 , wherein said processing comprises warping a representation of the face into rectangles using a piece-wise linear warping procedure, and dividing rectangles into a grid containing the plurality of face regions.

15. The method according to claim 14 , the selected subset of the plurality of face regions being further selected to be robust to facial movements and expressions, while being sufficiently discriminant to account for changes in skin color responsive to cardiac cycle variation for said computing the heart rate.

16. The method according to claim 13 , wherein said computing chrominance features comprises:
for each pixel, computing a chrominance signal C as a linear combination of two signals X f and Y f , such that C=X f −αY f , where

α
=

σ
⁡

(

X
f

)

σ
⁡

(

Y
f

)

 and σ(X f ), σ(Y f ) denote the standard deviations of X f , Y f ;
band-pass filtering signals the signals X and Y to obtain X f , Y f respectively, where X=3R n −2G n , Y=1.5R n +G n −1.5B n and R n , G n and B n are the normalized values of the individual color channels, wherein the color combination coefficients to derive X and Y are computed using a skin-tone standardization approach;
and, for each region r=1, . . . ,R, computing the final chrominance features averaging the values of the chrominance signals over all the pixels.

17. The method according to claim 13 , wherein said jointly estimating comprises enforcing a detection of chrominance feature variations that occur within a heart-rate frequency range.

18. The method according to claim 13 , wherein said jointly estimating comprises masking extracted regions of the plurality of face regions dependent on at least facial movement dependent changes.

19. The method according to claim 13 , wherein said jointly estimating comprises determining a local standard deviation over time of each extracted region of the plurality of face regions.

20. The method according to claim 13 , wherein said jointly estimating comprises employing, by the at least one automated processor, an alternating direction method of multipliers (ADMM), which solves an optimization problem by alternating a direction of the optimization while keeping other directions fixed.

21. The method according to claim 20 , wherein the solving of the optimization problem comprises repetitively performing the following three steps until convergence:
E/M-step
with fixed F and Z, obtaining optimal values of E and M by solving:

min
E

⁢

v
⁢


E


*

Assignees

• Univ Degli Studi Di Trento Univ Of Trento
• Fondazione Bruno Kessler Bruno Kessler Found
• Univ New York State Res Found
• Univ Of Pittsburgh—Of The Commonwealth Of Higher Education
• Univ Degli Studi Di Trento
• Fond Bruno Kessler
• Univ Of Pittsburgh Of The Commonwealth Of Higher Education

Inventors

• Sebe Niculae
• Alameda-Pineda Xavier
• Tulyakov Sergey
• Ricci Elisa
• Yin Lijun
• Cohn Jeffrey F

Classifications

• G06V10/56

  relating to colour · CPC title

• G06K9/00315

  Physics · mapped topic

• G06T2207/10016

  Video; Image sequence · CPC title

• A61B5/0077

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title

• A61B5/165

  Evaluating the state of mind, e.g. depression, anxiety · CPC title