Driver assistance based on pose detection

Patent metadata

Field	Value
Publication number	US-12559088-B2
Application number	US-202318240921-A
Country	US
Kind code	B2
Filing date	Aug 31, 2023
Priority date	Aug 31, 2023
Publication date	Feb 24, 2026
Grant date	Feb 24, 2026

Abstract

A video of a driver of a vehicle is obtained. Based on subset of frames of the video, a series of body poses are identified. The series of body poses are identified by detecting landmark points associated with respective body parts of the driver. The landmark points correspond to coordinates of locations of pixels that represent at least one or more joints of the respective body parts of the driver in the subset of frames. A driver behavior is identified based on the series of the body poses. An assistive vehicle control action for the vehicle is output based on the driver behavior.

First claim

What is claimed is:

1 . A method, comprising:
obtaining, using a camera facing a driver of a vehicle, a video of the driver in real-time during vehicle operation; identifying a series of body poses, wherein each of the body poses is identified based on respective first subset of frames of the video by detecting landmark points associated with respective body parts of the driver, wherein the landmark points correspond to coordinates of joint positions; determining changes in the series of body poses, wherein the changes correspond to combined Euclidean distance differences between corresponding landmark points of corresponding body parts in adjacent frames; applying power transformation to the changes to obtain transformed data; obtaining filtered data by applying a high-pass filter to the transformed data such that frequencies of signals representing slow changes in the series of the body poses that are less than a pre-determined threshold frequency are attenuated; identifying a driver behavior based on the filtered data; and automatically controlling the vehicle based on the driver behavior.

2 . The method of claim 1 , wherein the landmark points include coordinates representing joints of head, neck, torso, upper limbs, lower limbs, and spinal column of the driver.

3 . The method of claim 1 , wherein applying the power transformation comprises:
squaring the Euclidean distance differences, and wherein the high-pass filter attenuates frequencies less than a pre-determined threshold frequency.

4 . The method of claim 1 , the method further comprising:
extracting the video into segments responsive to determining that changes in the series of the body poses meet or exceed a pre-determined threshold condition, wherein the segments comprise the respective first subset of the frames.

5 . The method of claim 1 , the method further comprising:
obtaining biometric data of the driver; and determining a stress level of the driver based on a correlation between the biometric data and the driver behavior.

6 . The method of claim 5 , wherein the biometric data correspond to at least one of galvanic skin response data, heartrate data, electroencephalogram data, or functional magnetic resonance imaging data.

7 . The method of claim 1 , wherein the landmark points correspond to three-dimensional coordinates including x, y, and z values for each joint position.

8 . A vehicle comprising:
one or more sensors; a memory; and a processor configured to execute instructions stored in the memory to: obtain, using a camera facing a driver of the vehicle, a video of the driver in real-time during vehicle operation; identify a series of body poses, wherein each of the body poses is identified based on respective first subset of frames of the video by detecting landmark points associated with respective body parts of the driver, wherein the landmark points correspond to coordinates of joint positions; determine changes in the series of body poses, wherein the changes correspond to combined Euclidean distance differences between corresponding landmark points of corresponding body parts in adjacent frames; apply power transformation to the changes to obtain transformed data; obtain filtered data by applying a high-pass filter to the transformed data such that frequencies of signals representing slow changes in the series of the body poses that are less than a pre-determined threshold frequency are attenuated; identify a driver behavior based on the filtered data; and automatically control the vehicle based on the driver behavior.

9 . The vehicle of claim 8 , wherein the landmark points include coordinates representing joints of head, neck, torso, upper limbs, lower limbs, and spinal column of the driver.

10 . The vehicle of claim 9 , wherein to apply the power transformation comprises to: square the Euclidean distance differences, and wherein the high-pass filter attenuates frequencies less than a pre-determined threshold frequency.

11 . The vehicle of claim 9 , wherein the processor is further configured to execute instructions stored in the memory to:
obtain biometric data of the driver; and determine a stress level of the driver based on a correlation between the biometric data and the driver behavior.

12 . The vehicle of claim 11 , wherein the biometric data corresponds to at least one of galvanic skin response data, heartrate data, electroencephalogram data, or functional magnetic resonance imaging data.

13 . The vehicle of claim 8 , wherein the processor is further configured to execute instructions stored in the memory to:
extract the video into segments responsive to determining that changes in the series of the body poses meet or exceed a pre-determined threshold condition, wherein the segments comprise the respective first subset of the frames.

14 . The vehicle of claim 8 , wherein the landmark points correspond to three-dimensional coordinates including x, y, and z values for each joint position.

15 . A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations comprising:
obtaining, using a camera facing a driver of a vehicle, a video of the driver in real-time during vehicle operation; identifying a series of body poses, wherein each of the body poses is identified based on respective first subset of frames of the video by detecting landmark points associated with respective body parts of the driver, wherein the landmark points correspond to coordinates of joint positions; determining changes in the series of body poses, wherein the changes correspond to combined Euclidean distance differences between corresponding landmark points of corresponding body parts in adjacent frames; applying power transformation to the changes to obtain transformed data; obtaining filtered data by applying a high-pass filter to the transformed data such that frequencies of signals representing slow changes in the series of the body poses that are less than a pre-determined threshold frequency are attenuated; identifying a driver behavior based on the filtered data; and automatically controlling the vehicle based on the driver behavior.

16 . The non-transitory computer readable medium of claim 15 , wherein the landmark points include coordinates representing joints of head, neck, torso, upper limbs, lower limbs, and spinal column of the driver.

17 . The non-transitory computer readable medium of claim 15 , wherein applying the power transformation comprises:
squaring the Euclidean distance differences, and wherein the high-pass filter attenuates frequencies less than a pre-determined threshold frequency.

18 . The non-transitory computer readable medium of claim 15 , the operations further comprising:
extracting the video into segments responsive to determining that changes in the series of the body poses meet or exceed a pre-determined threshold condition, wherein the segments comprise the respective first subset of the frames.

19 . The non-transitory computer readable medium of claim 15 , the operations further comprising:
obtaining biometric data of the driver; and determining a stress level of the driver based on a correlation between the biometric data and the driver behavior.

20 . The non-transitory computer readable medium of claim 15 , wherein the landmark points correspond to three-dimensional coordinates including x, y, and z values for each joint position.

Assignees

• Nissan North America Inc

Inventors

• Heath Corey
• Witwicki Stefan

Classifications

• G06V20/49 Primary

  Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes · CPC title

• G06T2207/30268

  Vehicle interior · CPC title

• G06T2207/30196

  Human being; Person · CPC title

• B60W2540/221

  Physiology, e.g. weight, heartbeat, health or special needs · CPC title

• B60W2540/22

  Psychological state; Stress level or workload · CPC title