Control system and method

Patent metadata

Field	Value
Publication number	US-11065958-B2
Application number	US-201715397469-A
Country	US
Kind code	B2
Filing date	Jan 3, 2017
Priority date	Jan 3, 2017
Publication date	Jul 20, 2021
Grant date	Jul 20, 2021

Abstract

A control system having one or more controllers configured to determine physical or psychophysiological (3P) changes of an operator of a vehicle. First and second imaging devices take real-time images of an operator of a vehicle. Then, based on the physical, physiological and/or psychological features extracted from the imaging device data, and a 3P model from historical data the one or more processors also configured to, responsive to the physical, physiological and/or psychological changes of the operator of the vehicle, alert the operator and control the operation of the vehicle.

First claim

What is claimed is:

1. A control system comprising:
one or more processors configured to receive a first image from a first imaging device and determine physical, physiological or psychological changes of an operator of a vehicle without requiring intervention of the operator based on the first image received from the first imaging device; and
the one or more processors also configured to alert the operator responsive to the physical, physiological or psychological changes of the operator of the vehicle indicating that the operator is inattentive to control of the vehicle;
wherein the one or more processors are configured to receive a second image from one or more of a thermal imaging device, a midwave infrared (MWIR) camera, or a long wave infrared (LWIR) camera, and determine thermal changes related to the operator based on the second image;
wherein the one or more processors are configured to determine the physical, physiological or psychological changes of the operator using a new joint image that is generated by warping the second image onto the first image using a homography; and
wherein one or more pixels of the new joint image of the operator has red (R), green (G), blue (B), and thermal (T) values.

2. The control system of claim 1 , wherein the first imaging device is one or more of a digital camera comprising a detector array with one of red, green and blue wavelengths, a near infrared (NIR) camera, a short-wave infrared (SWIR) camera, a mid-microwave ultra-wide-band device, or a depth measuring camera.

3. The control system of claim 1 , wherein the first image from the first imaging device and the second image from the thermal imaging device are both of a face of the operator.

4. The control system of claim 3 , wherein the one or more processors also are configured to determine a probability of observing a facial cue.

5. The control system of claim 1 , wherein the one or more processors are configured to alert the operator by actuating an alarm.

6. The control system of claim 1 , wherein the one or more processors are configured to alert the operator by automatically engaging a brake of the vehicle.

7. The control system of claim 1 , wherein the one or more processors are configured to form a model of the operator related to physical, physiological and psychological parameters of the operator based on the physical, physiological or psychological changes of the operator or a combination of the physical, physiological or psychological changes of the operator.

8. The control system of claim 7 , wherein the physical, physiological and psychological parameters are measured by one or more of an electroencephalographic (EEG) device, a photoplethysmographic (PPG) device, an electrocardiographic (ECG) device, a ballistocardiographic (BCG) device, an electromyographic (EMG) device, a phonocardiographic (PCG) device, a galvanic skin response (GSR) device, a transcranial Doppler (TCD) sonography device, or functional near infrared (fNIR) signals, on a human in a controlled setting.

9. The control system of claim 1 , wherein the one or more processors also are configured to warp the new joint image into a canonical frontal view.

10. The control system of claim 1 , wherein the one or more processors also are configured to:
determine the operator is incapacitated based on the physical, physiological or psychological changes of the operator determined; and
stop the vehicle responsive to determining the operator is incapacitated.

11. The control system of claim 1 , wherein the one or more processors also are configured to:
responsive to the physical, physiological or psychological changes of the operator of the vehicle that are determined indicating that the operator is inattentive to control of the vehicle, alert a remote operations center.

12. The control system of claim 1 , wherein the physical, physiological or psychological changes of the operator include eye movement away from a front of the vehicle.

13. A method comprising:
receiving a first image from a first imaging device;
determining physical, physiological or psychological changes of an operator of a vehicle based on the first image received from the first imaging device;
alerting the operator responsive to the physical, physiological or psychological changes of the operator of the vehicle indicating that the operator is inattentive to control of the vehicle;
receiving a second image from one of a thermal imaging device, a midwave infrared (MWIR) camera, or a long wave infrared (LWIR) camera, and determine thermal changes related to the operator based on the second image received from the thermal imaging device;
determining the physical, physiological or psychological changes of the operator using a new joint image that is generated by warping the second image onto the first image using a homography; and
wherein one or more pixels of the new joint image of the operator has red (R), green (G), blue (B), and thermal (T) values.

14. The method of claim 13 further comprising the step of recording the operator with a first imaging device to collect information related to the operator used to determine the physical, physiological or psychological changes of the operator.

15. The method of claim 13 further comprising the step of comparing the physical, physiological or psychological changes of the operator to historical data to determine when to alert the operator of the vehicle with the alarm.

16. The method of claim 15 wherein the historical data is a model.

17. The method of claim 15 wherein the historical data is a predetermined threshold for health parameters and fatigue and drowsiness levels.

18. A control system comprising:
one or more processors configured to determine physical, physiological or psychological changes of an operator of a vehicle;
the one or more processors also configured to brake the vehicle responsive to the physical, physiological or psychological changes of the operator of the vehicle indicating that the operator is inattentive to control the vehicle;
wherein the one or more processors are configured to receive a first image from a first imaging device and determine the physical, physiological or psychological changes of the operator based on the first image;
wherein the one or more processors are configured to receive a second image from one or more of a thermal imaging device, a midwave infrared (MWIR) camera, or a long wave infrared (LWIR) camera, and determine thermal changes related to the operator based on the second image;
wherein the one or more processors are configured to generate a new joint image by warping the second image onto the first image using a homography and to determine the physical, physiological or psychological changes of the operator based on the new joint image; and
wherein one or more of the pixels of the new joint image of the operator has red (R), green (G), blue (B), and thermal (T) values.

19. The control system of claim 18 wherein the one or more processors also are configured to first alert the operator responsive to the physical, physiological or psychological changes of the operator and then brake the vehicle based on a lack of response by the operator to the alert.

20. The control system of claim 19 wherein the one or more processors also are configured to receive information from at least one imaging device monitoring a health parameter of the operator.

21. The control system of claim 20 wherein the one or more processors also are configured to receive inform

Assignees

• Gen Electric
• Transp Ip Holdings Llc

Inventors

• Mestha Lalit Keshav
• Brooks James
• Tu Peter

Classifications

• G06V20/597

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title

• G06V10/147 Primary

  Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19) · CPC title

• G06V10/143

  Sensing or illuminating at different wavelengths · CPC title

• A61B5/384

  Recording apparatus or displays specially adapted therefor · CPC title

• G06V40/176

  Dynamic expression · CPC title