Methods and vehicles for capturing emotion of a human driver and moderating vehicle response

Patent metadata

Field	Value
Publication number	US-10453453-B2
Application number	US-201715787677-A
Country	US
Kind code	B2
Filing date	Oct 18, 2017
Priority date	Apr 22, 2011
Publication date	Oct 22, 2019
Grant date	Oct 22, 2019

Abstract

Methods and systems for determining an emotion of a human driver of a vehicle and using the emotion for generating a vehicle response, is provided. One example method includes capturing, by a camera of the vehicle, a face of the human driver. The capturing is configured to capture a plurality of images over a period of time, and the plurality of images are analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time. The method further includes capturing, by a microphone of the vehicle, voice input of the human driver. The voice input is captured over the period of time. The voice input is analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time. The method processes, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to predict the emotion of the human driver. The method generates the vehicle response that is responsive to the emotion of the human driver. The vehicle response is configured to make at least one adjustment to a setting of the vehicle. The adjustment is selected based on the emotion of the human driver. The vehicle response can be used to make the driver more calm and/or assist in reducing distracted driving. The prediction of the emotion may be additionally increased by capturing and analyzing touch and/or gesture characteristic of the human driver when interfacing with a graphical user interface or surfaces of the vehicle or systems of the vehicle.

First claim

What is claimed is:

1. A method for determining an emotion of a human driver of a vehicle and using the emotion for generating a vehicle response, comprising,
capturing, by a camera of the vehicle, a face of the human driver, the capturing is configured to capture a plurality of images over a period of time, the plurality of images being analyzed to identify a facial expression and changes in the facial expression of the human driver over the period of time;
capturing, by a microphone of the vehicle, voice input of the human driver, the voice input is captured over the period of time, the voice input being analyzed to identify a voice profile and changes in the voice profile of the human driver over the period of time;
processing, by a processor of the vehicle, a combination of the facial expression and the voice profile captured during the period of time to predict the emotion of the human driver; and
generating the vehicle response that is responsive to the emotion of the human driver, the vehicle response is configured to make at least one adjustment to a setting of the vehicle, the adjustment is selected based on the emotion of the human driver.

2. The method of claim 1 , wherein combining data obtained from facial expressions and the voice profile improve the prediction of the emotion of the human driver and wherein prediction of the emotion is increased by capturing touch or gesture characteristic of the human driver when interfacing with a graphical user interface of the vehicle or systems of the vehicle.

3. The method of claim 1 , wherein facial characteristics of the human driver is are related to a plurality of facial expressions that have been previously associated with specific emotions, the facial characteristics include one or more of placement of the human driver's mouth, a way eyes are opened or closed by the human driver, cheekbone characteristics of the human driver, or biometric determining of features of the human driver, or a combination of two or more thereof.

4. The method of claim 1 , wherein processing the voice profile includes processing an audio sample of the voice input and identifying frequency and/or magnitude characteristics in the audio sample, the voice profile is associated with a tone of voice that is correlated to a possible emotion among a plurality of emotions.

5. The method of claim 1 , wherein facial characteristics of the human driver is are related to a plurality of facial expressions that have been previously associated with specific emotions, the facial characteristics include one or more of placement of the human driver's mouth, a way eyes are opened or closed by the human driver, cheekbone characteristics of the human driver, or biometric determining of features of the human driver, or a combination of two or more thereof and the processing the voice profile includes processing an audio sample of the voice input and identifying frequency and/or magnitude characteristics in the audio sample, the voice profile is associated with a tone of voice that is correlated to a possible emotion among a plurality of emotions.

6. The method of claim 5 , wherein the plurality of facial expressions are associated to a plurality of templates of facial characteristics, the plurality of templates being generated during a training or calibration session or generated over time based on learned facial characteristics, wherein learned facial characteristics are identified using one or more other detected physical or physiological conditions.

7. The method of claim 6 , wherein said physiological conditions include one or more of heart rate, or body heat, or blood pressure, or skin temperature, or skin galvanic resistance, or temperature of skin, or blood rush to a face of the human driver, or eye iris or retina changes, or odor/scent changes, or combinations of two or more thereof, and the physical conditions include one or more of a changes in the face of the human driver, or changes in a typing rhythm, or changes in gait, or changes in voice, changes in voice tone, or voice inflections, or voice speed or slowness, or sharp voice harmonics, or voice identifiers or patterns, or elevated heat patterns, or palm veins, or eye iris or retina changes, heat rate changes, or eye gaze patterns or motions, or combinations of two or more thereof.

8. The method of claim 1 , wherein a user account with a profile for the human driver of the vehicle is provided, the profile includes a plurality of learned facial expressions and learned voice profiles that are indicative of an emotion, the learned plurality of facial expressions and learned voice profiles are updated from time to time to improve the prediction of the emotion of the human driver, wherein the user profile is synchronized from time to time with a server that manages the user account for the human driver.

9. The method of claim 1 , wherein the vehicle response to the emotion includes one of waking up the human driver with a sound or air if the emotion is a sleepy emotion, generating warnings and/or signals to alert the human driver, generating recommendations to calm the human driver from an angry emotion or agitated emotion, sending a notification to a third party indicating that the human driver is tired, generating a notification to the human driver suggesting a temperature change, or generating an automatic temperature change based on the emotion, recommending the human driver stop driving when tired, reducing recommendations when the human driver when the predicted emotion is a rushed emotion, changing lighting of the vehicle automatically, changing temperature of the vehicle automatically, turn up or down a volume of music, adjust a seat position, or a combination of two or more thereof, and wherein at least one of said vehicle responses are predefined to reduce distracted driving or increase alertness of said human driver.

10. The method of claim 1 , wherein the vehicle response is further based on current use of the vehicle, and use of the vehicle includes one or more of geolocation of the vehicle, speed of the vehicle, direction of the vehicle, a route of the vehicle, a time of day, occupants in the vehicle, historical use of the vehicle, online data of a user accessible to the vehicle or server, or combinations of two or more thereof, and the vehicle response to the emotion includes one of waking up the human driver with a sound or air if the emotion is a sleepy emotion, generating warnings and/or signals to alert the human driver, generating recommendations to calm the human driver from an angry emotion or agitated emotion, sending a notification to a third party indicating that the human driver is tired, generating a notification to the human driver suggesting a temperature change, or generating an automatic temperature change based on the emotion, recommending the human driver stop driving when tired, reducing recommendations when the human driver when the predicted emotion is a rushed emotion, or a combination of two or more thereof.

11. The method of claim 1 , wherein a user account with a profile for the human driver of the vehicle is provided, wherein the user account of the human driver is identified based on processing of credential data obtained from the human driver, the credential data used by electronics of the vehicle to associated the human driver to the user account and to the profile, the electronics of the vehicle configured to interface over a network with a server for use of the profile and associated updates to the profile as made via use of the vehicle or updates associated with learned behavior.

12. The method of claim 1 , wherein the emotion includes one or more of a normal emotion, a frustrated

Assignees

• Emerging Automotive Llc

Inventors

• Penilla Angel A
• Penilla Albert S

Classifications

• G01C21/3641

  Personalized guidance, e.g. limited guidance on previously travelled routes · CPC title

• G10L25/63

  for estimating an emotional state · CPC title

• H04L67/10

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• H04L67/306

  User profiles · CPC title

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title