System and method for remote measurements of vital signs of a person in a volatile environment

Patent metadata

Field	Value
Publication number	US-12056870-B2
Application number	US-202117199696-A
Country	US
Kind code	B2
Filing date	Mar 12, 2021
Priority date	May 16, 2018
Publication date	Aug 6, 2024
Grant date	Aug 6, 2024

Abstract

A remote photoplethysmography (RPPG) system for estimating vital signs of a person is provided. The RPPG system is configured to receive a set of imaging photoplethysmography (iPPG) signals measured from different regions of a skin of a person. The RPPG system is further configured to determine frequency coefficients at the frequency bins of the quantized frequency spectrum of the measured iPPG signals by minimizing a distance between the measured iPPG signals and corresponding iPPG signals reconstructed from the determined frequency coefficients, while enforcing joint sparsity of the determined frequency coefficients subject to the sparsity level constraint, such that the determined frequency coefficients of different iPPG signals have the non-zero values at the same frequency bins; and output one or a combination of the determined frequency coefficients, the iPPG signals reconstructed from the determined frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signals.

First claim

The invention claimed is:

1. A remote photoplethysmography (RPPG) system for estimating vital signs of a person, comprising: at least one processor; and memory having instructions stored thereon that, when executed by the at least one processor, cause the RPPG system to:
receive a set of imaging photoplethysmography (iPPG) signals measured from different regions of a skin of a person;
determine a sparsity level constraint indicative of a number of frequency bins of a quantized frequency spectrum of the measured iPPG signals having non-zero values of frequency coefficients;
determine frequency coefficients at the frequency bins of the quantized frequency spectrum of the measured iPPG signals by minimizing a distance between the measured iPPG signals and corresponding iPPG signals reconstructed from the determined frequency coefficients, while enforcing joint sparsity of the determined frequency coefficients subject to the sparsity level constraint, such that the determined frequency coefficients of different iPPG signals have the non-zero values at the same frequency bins; and
output one or a combination of the determined frequency coefficients, the iPPG signals reconstructed from the determined frequency coefficients, and a vital sign signal corresponding to the reconstructed iPPG signals.

2. The RPPG system of claim 1 , wherein the sparsity level constraint is determined adaptively based on a function of intensities in the measured iPPG signals as a bound on the minimum energy that a jointly sparse signal embedded in the measured iPPG signals can hold, and wherein the sparsity level constraint enforces an upper bound on energy levels of the determined frequency coefficients of the reconstructed iPPG signals.

3. The RPPG system of claim 2 , wherein the measured iPPG signals are subject to measurement noise, and wherein the bound on the minimum energy of the jointly sparse signal embedded in the measured iPPG signals is determined iteratively by minimizing energy deviation based on a gradient of the distance of the reconstructed iPPG signals to the measured iPPG signals, wherein the gradient is computed with respect to the frequency coefficients and the measurement noise.

4. The RPPG system of claim 1 , wherein the processor is further configured to execute a two-one norm regularization of a frequency matrix of the frequency coefficients determined for reconstructing the iPPG signals modified with a noise matrix of measurement noise, and wherein the two-one norm regularization is applied to opposite dimensions of the frequency matrix and the noise matrix.

5. The RPPG system of claim 4 , wherein a first dimension of the frequency matrix corresponds to the different regions of the skin of the person, wherein a second dimension of the frequency matrix corresponds to frequency bins of the frequency coefficients, wherein a first dimension of the noise matrix corresponds to the different regions of the skin of the person, wherein a second dimension of the noise matrix corresponds to a time of the measurements, and wherein the two-one norm regularization is applied to opposite dimensions such that two-norm along the first dimension of the frequency matrix is followed by a one-norm along the second dimension of the frequency matrix, while a two-norm along the second dimension of the noise matrix is followed by a one-norm along the first dimension of the noise matrix.

6. The RPPG system of claim 1 , wherein the sparsity level constraint is enforced by a regularization parameter that is determined iteratively in response to updated estimates of the reconstructed iPPG signals to ensure that energy of the frequency coefficients of the reconstructed iPPG signals equals the sparsity level constraint.

7. The RPPG system of claim 1 , wherein the processor is further configured to:
determine weights of the frequency bins indicative of which of the frequency bins have the frequency coefficients with the non-zero values based on a function of the phase differences across the measured iPPG signals; and
enforce the joint sparsity by encouraging the number of non-zero frequency coefficients indicated by the sparsity level constraint to be at the locations indicated by the weights of the frequency bins.

8. The RPPG system of claim 7 , wherein the processor is further configured to execute a weighted two-one norm regularization of a frequency matrix of the frequency coefficients determined for reconstructing the iPPG signals modified with a noise matrix of measurement noise, and wherein the weighted two-one norm regularization is applied to opposite dimensions of the frequency matrix and the noise matrix.

9. The RPPG system of claim 8 , wherein a first dimension of the frequency matrix corresponds to the different regions of the skin of the person, wherein a second dimension of the frequency matrix corresponds to frequency bins of the frequency coefficients, wherein a first dimension of the noise matrix corresponds to the different regions of the skin of the person, wherein a second dimension of the noise matrix corresponds to a time of the measurements, and wherein the weighted two-one norm regularization is applied to opposite dimensions such that two-norm along the first dimension of the frequency matrix is followed by a weighted one-norm along the second dimension of the frequency matrix, while a two-norm along the second dimension of the noise matrix is followed by a weighted one-norm along the first dimension of the noise matrix.

10. The RPPG system of claim 9 , wherein weights in the weighted one-norm along the first dimension of the noise matrix are identical.

11. The RPPG system of claim 9 , wherein weights in the weighted one-norm along the second dimension of the frequency matrix are identical, and wherein weights in the weighted one-norm along the first dimension of the noise matrix are identical.

12. The RPPG system of claim 9 , wherein the weights in the weighted one-norm along the second dimension of the frequency matrix are a function of the phase differences across the measured iPPG signals from different regions.

13. The RPPG system of claim 1 , wherein to obtain the measured iPPG signals, the processor is further configured to:
receive a set of iPPG signals measured from a set of skin regions of the person;
group the set of iPPG signals into median regions to produce a clustering of iPPG signals; and
compute a measured iPPG signal for each median region that is a median across the iPPG signals measured from the skin regions that form the median region.

14. The RPPG system of claim 13 , wherein the processor is further configured to remove iPPG signals of a region from the set of iPPG signals when energy of the measured iPPG signals for the region within a time window is above a threshold.

15. The RPPG system of claim 13 , wherein the iPPG signals in the set of iPPG signals are noisy due to measurement noise, and wherein the processor is further configured to denoise the noisy iPPG signals in the set of iPPG signals by projecting the noisy iPPG signals onto an orthogonal complement of a noise subspace.

16. The RPPG system of claim 15 , wherein to project the noisy iPPG signals P onto the orthogonal complement of the noise subspace Q, the processor is further configured to orthogonally project the noisy iPPG signals P onto the noise subspace Q to produce an orthogonal projection of the noisy iPPG signals, and then subtract the orthogonal projection of the noisy iPPG signals from the noisy iPPG signals P to produce denoised iPPG signals Z.

17. The RPPG system of claim 15 , wherein the noise su

Assignees

• Mitsubishi Electric Res Laboratories Inc
• Mitsubishi Electric Corp

Inventors

• Marks Tim
• Mansour Hassan
• Nowara Ewa
• NAKAMURA Yudai
• Veeraghavan Ashok

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T5/20

  using local operators · CPC title

• B60W30/182

  Selecting between different operative modes, e.g. comfort and performance modes · CPC title

• B62D6/00

  Arrangements for automatically controlling steering depending on driving conditions sensed and responded to, e.g. control circuits (means for initiating a change in direction B62D1/00; steering valves B62D5/06; combined with means for inclining the vehicle body or wheels on bends B62D9/00) · CPC title

• B60T7/12

  for automatic initiation; for initiation not subject to will of driver or passenger {(limiting speed of vehicles other than rail vehicles B60K31/00)} · CPC title