Method, apparatus and device for identifying passenger state in unmanned vehicle, and storage medium

Patent metadata

Field	Value
Publication number	US-11315349-B2
Application number	US-201916516137-A
Country	US
Kind code	B2
Filing date	Jul 18, 2019
Priority date	Sep 10, 2018
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

Embodiments of the present application provide a method, an apparatus, and a device for identifying a passenger state in an unmanned vehicle, and a storage medium. The method comprises: obtaining monitoring data of different dimensions in a process where the passenger takes the unmanned vehicle; performing feature extraction on the monitoring data of the different dimensions and forming feature data of different dimensions; and identifying the passenger state according to the feature data of the different dimensions. By obtaining the monitoring data of various dimensions in the process where the passenger takes the unmanned vehicle to identify the passenger state, it is possible to omnidirectionally monitor the personal is safety and property safety of the passengers, and effectively protect the passenger taking the unmanned vehicle.

First claim

What is claimed is:

1. A method for identifying a passenger state in an unmanned vehicle, comprising:
obtaining monitoring data of different dimensions in a process where a passenger takes the unmanned vehicle, wherein the monitoring data of different dimensions comprises expression monitoring data, limb movement monitoring data, sound monitoring data, vital sign monitoring data, and collision data of colliding a vehicle body or a seat;
performing feature extraction on the monitoring data of different dimensions and forming feature data of different dimensions;
identifying the passenger state according to the feature data of different dimensions, wherein the passenger state comprises a safety state and a danger state; and
executing at least one step of:
reporting the danger state, identity information of the passenger and contact information of the passenger's family to a server, if the passenger state is the danger state, for the server to notify the passenger's family according to the identity information of the passenger and the contact information of the passenger's family, wherein the identity information of the passenger and the contact information of the passenger's family are pre-stored in the unmanned vehicle; and
reporting position of the unmanned vehicle to the server, if the passenger's state is the danger state, for the server to obtain the position of the unmanned vehicle that the passenger takes, wherein the unmanned vehicle is located by a GPS module provided on the unmanned vehicle,
wherein the identifying the passenger state according to the feature data of different dimensions comprises:
inputting feature data of each dimension into a corresponding first identification algorithm, and outputting passenger state probability data corresponding to the feature data of each dimension;
obtaining a weight value corresponding to the passenger state probability data of each dimension;
performing a weighted summation calculation on passenger state probability data of respective dimensions to obtain general passenger state probability data; and
determining the passenger state according to the general passenger state probability data and a preset threshold.

2. The method according to claim 1 , wherein the obtaining monitoring data of different dimensions in a process where the passenger takes the unmanned vehicle comprises:
collecting in real time, by different sensors provided in the unmanned vehicle, monitoring data of corresponding dimensions in the process where the passenger takes the unmanned vehicle.

3. The method according to claim 2 , wherein the sensors comprise at least: an internal camera, a microphone, a vital sign sensor, and a collision sensor.

4. The method according to claim 3 , wherein the collecting in real time, by different sensors provided in the unmanned vehicle, monitoring data of corresponding dimensions in the process where the passenger takes the unmanned vehicle, comprises:
collecting, by the internal camera, the expression monitoring data and the limb movement monitoring data in the process where the passenger takes the unmanned vehicle;
collecting, by the microphone, the sound monitoring data in the process where the passenger takes the unmanned vehicle;
collecting, by the vital sign sensor, the vital sign monitoring data in the process where the passenger takes the unmanned vehicle; and
collecting, by the collision sensor, the collision data of colliding a vehicle body or a seat in the process where the passenger takes the unmanned vehicle.

5. The method according to claim 1 , wherein the performing feature extraction on the monitoring data of different dimensions and forming feature data of different dimensions comprises:
performing the feature extraction on the monitoring data of each dimension by means of a corresponding feature extraction algorithm and forming feature data of a corresponding dimension.

6. An apparatus for identifying a passenger state in an unmanned vehicle, comprising:
a processor and a computer-readable medium for storing program codes, which, when executed by the processor, cause the processor to: obtain monitoring data of different dimensions in a process where a passenger takes the unmanned vehicle, wherein the monitoring data of different dimensions comprises expression monitoring data, limb movement monitoring data, sound monitoring data, vital sign monitoring data, and collision data of colliding a vehicle body or a seat;
perform feature extraction on the monitoring data of different dimensions and form feature data of different dimensions;
identify the passenger state according to the feature data of different dimensions, wherein the passenger state comprises a safety state and a danger state; and
execute at least one step of:
reporting the danger state, identity information of the passenger and contact information of the passenger's family to a server, if the passenger state is the danger state, for the server to notify the passenger's family according to the identity information of the passenger and the contact information of the passenger's family, wherein the identity information of the passenger and the contact information of the passenger's family are pre-stored in the unmanned vehicle; and
reporting position of the unmanned vehicle to the server, if the passenger's state is the danger state, for the server to obtain the position of the unmanned vehicle that the passenger takes, wherein the unmanned vehicle is located by a GPS module provided on the unmanned vehicle,
wherein the program codes further cause the processor to:
input feature data of each dimension into a corresponding first identification algorithm, and output passenger state probability data corresponding to the feature data of each dimension;
obtain a weight value corresponding to the passenger state probability data of each dimension;
perform a weighted summation calculation on passenger state probability data of respective dimensions to obtain general passenger state probability data; and
determine the passenger state according to the general passenger state probability data and a preset threshold.

7. The apparatus according to claim 6 , wherein the program codes further cause the processor to:
collect in real time, by different sensors provided in the unmanned vehicle, monitoring data of corresponding dimensions in the process where the passenger takes the unmanned vehicle.

8. The apparatus according to claim 7 , wherein the sensors comprise at least:
an internal camera, a microphone, a vital sign sensor, a collision sensor.

9. The apparatus according to claim 8 , wherein the program codes further cause the processor to:
collect, by the internal camera, the expression monitoring data and the limb movement monitoring data in the process where the passenger takes the unmanned vehicle;
collect, by the microphone, the sound monitoring data in the process where the passenger takes the unmanned vehicle;
collect, by the vital sign sensor, the vital sign monitoring data in the process where the passenger takes the unmanned vehicle; and
collect, by the collision sensor, the collision data of colliding a vehicle body or a seat in the process where the passenger takes the unmanned vehicle.

10. The apparatus according to claim 6 , wherein the program codes further cause the processor to:
perform the feature extraction on the monitoring data of each dimension by means of a corresponding feature extraction algorithm and form feature data of a corresponding dimension.

11. A terminal device, comprising:
one or more processors;
wherein one or more programs, when executed by the one or more processors, cause the one or more proc

Assignees

• Apollo Intelligent Driving Tech Beijing Co Ltd

Inventors

• Chen Youhan

Classifications

• G06V20/597 Primary

  Recognising the driver's state or behaviour, e.g. attention or drowsiness · CPC title

• G06V10/811

  the classifiers operating on different input data, e.g. multi-modal recognition · CPC title

• G06V20/59 Primary

  inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions · CPC title

• G06F18/256

  of results relating to different input data, e.g. multimodal recognition · CPC title

• G06F18/253

  of extracted features · CPC title