Optical heartrate tracking

Patent metadata

Field	Value
Publication number	US-9962095-B2
Application number	US-201313869016-A
Country	US
Kind code	B2
Filing date	Apr 23, 2013
Priority date	Apr 23, 2013
Publication date	May 8, 2018
Grant date	May 8, 2018

Abstract

Heartrate tracking is performed entirely optically without the subject being required to wear any monitoring equipment by processing a combination of signals representing frames of video of the sinusoidal motion of a subject's facial skin color changes captured by both IR and visible light (e.g., RGB—red/green/blue) cameras. The IR and RGB graphs that result from the processing are perfectly phase-shifted so that when the IR signal is going down in amplitude, the RGB signal is going up. Such phase-shifting enables the optical heartrate tracking to utilize diverse input feeds so that a tracked signal is accepted as the user's true heartrate when both IR and RGB signals are well correlated.

First claim

What is claimed:

1. One or more hardware-based computer-readable memory devices not consisting of propagated signals having computer readable instructions stored thereupon that, when executed by a computer, implement:
a face alignment and preparation pipeline configured to receive respective streams of infrared video frames and visible light video frames which include a facial image of a human subject and simultaneously process the received infrared video frames and visible light video frames to output a decomposed, frame-aligned, facial skin-only signal measurement comprising spatially-corresponding infrared light and visible light graphs that are phase-shifted so that as values in the infrared light graph are increasing, values in the spatially-corresponding visible light graph are decreasing, and vice-versa;
a signal-finding pipeline configured to: i) receive the decomposed, frame-aligned, facial skin-only signal measurement, ii) compare the signal measurement in a current frame to a cached decomposed version of a head image in a previous frame, iii) sum up and average pixels having deltas between head images in the current frame and previous frame that trend in a consistent direction, and are within a set number of standard deviations of brightness, iv) generate a measurement of noisiness per pixel, v) apply at least one filter to the summed and averaged pixels, the filter being weighted by the noisiness measurement, vi) create a time-space representation of low-amplitude deltas in a skin-only facial image;
a signal-extracting pipeline configured to a) apply time-space smoothing or frequency-space smoothing to the time-space representation, b) apply a discrete Fourier transform to transform the time-space representation into a frequency-space representation for each of the infrared and visible streams, c) merge the frequency-space representations weighted by graph noise level; and
a heartrate identification pipeline configured to identify a heartrate of the human subject from the merged frequency-space representations.

2. The one or more hardware-based computer-readable memory devices of claim 1 in which the heartrate identification pipeline is configured to identify a highest peak in the merged frequency-space graphs.

3. The one or more hardware-based computer-readable memory devices of claim 2 in which the pipelines operate sequentially on a given video frame.

4. The one or more hardware-based computer-readable memory devices of claim 2 in which the pipelines operate in parallel.

5. The one or more hardware-based computer-readable memory devices of claim 2 further comprising configuring the heartrate identification pipeline to measure a noise level of the highest peak.

6. The one or more hardware-based computer-readable memory devices of claim 5 further comprising configuring the heartrate identification pipeline to bucket a location of the highest peak, weighted by the noise level into a frequency detector.

7. The one or more hardware-based computer-readable memory devices of claim 1 further comprising configuring the heartrate identification pipeline to apply a smoothing factor to nearby signals and sum the signals together.

8. The one or more hardware-based computer-readable memory devices of claim 7 further comprising configuring the heartrate identification pipeline to output a graph of the summed, smoothed signals.

9. The one or more hardware-based computer-readable memory devices of claim 8 further comprising configuring the heartrate identification pipeline to identify a heartrate signal from the graph if a peak result is above a set amplitude threshold.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Ahmad Anis
• Walker Ben
• Kennett Daniel
• Flavell Andrew

Classifications

• A61B5/024 Primary

  Measuring pulse rate or heart rate · CPC title

• A61B5/0077 Primary

  Devices for viewing the surface of the body, e.g. camera, magnifying lens · CPC title

• A61B5/7257

  using Fourier transforms · CPC title

• A61B5/7207

  of noise induced by motion artifacts · CPC title

• A61B5/02416

  using photoplethysmograph signals, e.g. generated by infrared radiation (A61B5/14552 takes precedence) · CPC title