Vulnerable road user's adversarial behavior recognition

Patent metadata

Field	Value
Publication number	US-12280786-B2
Application number	US-202318319037-A
Country	US
Kind code	B2
Filing date	May 17, 2023
Priority date	May 17, 2023
Publication date	Apr 22, 2025
Grant date	Apr 22, 2025

Abstract

A system for identifying adversarial behavior directed to an autonomous vehicle includes a pedestrian detection system in communication with a vehicle controller adapted to identify a pedestrian within proximity of the autonomous vehicle, track the pedestrian, determine if the pedestrian is intending to cross in front of the autonomous vehicle, and if the pedestrian tracking system determines that the pedestrian is not intending to cross in front of the autonomous vehicle, the pedestrian tracking system is further adapted to continue tracking the pedestrian, and if the pedestrian tracking system determines that the pedestrian is intending to cross in front of the autonomous vehicle, the pedestrian tracking system is further adapted to determine, with an adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior.

First claim

What is claimed is:

1. A method of identifying adversarial behavior directed to an autonomous vehicle, comprising:
identifying, with a pedestrian detection system in communication with a vehicle controller, a pedestrian within proximity of the autonomous vehicle;
tracking the pedestrian with the pedestrian tracking system;
determining, with the pedestrian tracking system, if the pedestrian is intending to cross in front of the autonomous vehicle;
if the pedestrian tracking system determines that the pedestrian is not intending to cross in front of the autonomous vehicle, continuing tracking of the pedestrian with the pedestrian tracking system; and
if the pedestrian tracking system determines that the pedestrian is intending to cross in front of the autonomous vehicle, determining, with an adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior.

2. The method of claim 1 , further including:
if the pedestrian tracking system determines that the pedestrian is intending to cross in front of the autonomous vehicle, determining, with the vehicle controller, if the autonomous vehicle has right of way, and:
if the autonomous vehicle does not have the right of way, continuing tracking of the pedestrian with the pedestrian tracking system; and
if the autonomous vehicle does have the right of way, determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior.

3. The method of claim 2 , wherein the determining, with the vehicle controller, if the autonomous vehicle has a right of way further includes:
determining, with the vehicle controller, what type of crossing is present in front of the autonomous vehicle, to be used by the pedestrian;
identifying, with the vehicle controller, crossing signals for the identified crossing type; and
determining, with the vehicle controller, if the autonomous vehicle has the right of way based on the identified crossing signals.

4. The method of claim 1 , wherein the determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior further includes determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior based on at least one of spatiotemporally analyzing the crossing behavior of the pedestrian, non-verbal cues from the pedestrian and audible cues.

5. The method of claim 4 , wherein the determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior based on the crossing behavior of the pedestrian further includes:
calculating, with the vehicle controller, an estimated crossing time for the pedestrian to cross in front of the autonomous vehicle;
determining, with the adversarial intent algorithm, that the pedestrian is not exhibiting any adversarial behavior when the pedestrian finishes crossing in front of the autonomous vehicle within a predetermined time window; and
determining, with the adversarial intent algorithm, that the pedestrian is exhibiting adversarial behavior when the pedestrian does not finish crossing in front of the autonomous vehicle within the predetermined time window.

6. The method of claim 4 , wherein the determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior based on non-verbal cues from the pedestrian further includes:
identifying, with the pedestrian tracking system, physical properties of the pedestrian;
analyzing, with the vehicle controller, spatiotemporal characteristics of the physical properties of the pedestrian; and
determining, with the vehicle controller, if any physical properties of the pedestrian indicate adversarial behavior by the pedestrian.

7. The method of claim 6 , wherein the analyzing, with the vehicle controller, spatiotemporal characteristics of the physical properties of the pedestrian further includes:
identifying, with the pedestrian tracking system, key points of a hand of the pedestrian;
comparing, with the vehicle controller, identified key points of the hand of the pedestrian to a remote database of known threatening gestures; and
determining, with the adversarial intent algorithm, that the pedestrian is exhibiting adversarial behavior when the identified key points of the hand of the pedestrian match at least one of the known threatening gestures within the remote database.

8. The method of claim 4 , wherein the determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior based on audible cues further includes:
capturing, with an external microphone, acoustic signals from the pedestrian;
extracting, with the vehicle controller, features of the acoustic signals; and
comparing, with the vehicle controller, features of the acoustic signals to a remote database which includes an optimized model of known sounds that indicate adversarial behavior; and
determining, with the adversarial intent algorithm, that the pedestrian is exhibiting adversarial behavior when the captured acoustic signals match known sounds that indicate adversarial behavior.

9. The method of claim 4 , wherein, the determining, with the adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior based on at least one of crossing behavior of the pedestrian, non-verbal cues from the pedestrian and audible cues further includes using an information fusion algorithm within the vehicle controller to combine information related to the crossing behavior of the pedestrian, information related to non-verbal cues from the pedestrian and audible cues to identify adversarial behavior.

10. The method of claim 9 , wherein, when the pedestrian is exhibiting adversarial behavior, the method further includes:
classifying, with a persistence algorithm, the adversarial behavior as “no risk” when a duration of such adversarial behavior is less than a first pre-determined length of time;
classifying, with the persistence algorithm, the adversarial behavior as “low risk” when the duration of such adversarial behavior is greater than the first pre-determined length of time and less than a second pre-determined length of time; and
classifying, with the persistence algorithm, the adversarial behavior as “high risk” when the duration of such adversarial behavior is greater than the second pre-determined length of time.

11. A system for identifying adversarial behavior directed to an autonomous vehicle, comprising:
a pedestrian detection system in communication with a vehicle controller, adapted to:
identify a pedestrian within proximity of the autonomous vehicle;
track the pedestrian;
determine if the pedestrian is intending to cross in front of the autonomous vehicle; and:
if the pedestrian tracking system determines that the pedestrian is not intending to cross in front of the autonomous vehicle, the pedestrian tracking system is further adapted to continue tracking the pedestrian; and
if the pedestrian tracking system determines that the pedestrian is intending to cross in front of the autonomous vehicle, the pedestrian tracking system is further adapted to determine, with an adversarial intent algorithm, if the pedestrian is exhibiting any adversarial behavior.

12. The system of claim 11 , wherein, if the pedestrian tracking system determines that the pedestrian is intending to cross in front of the autonomous vehicle, the pedestrian tracking system is further adapted to determine, with the vehicle controller, if the autonomous vehicle has right of way, and:
if the autonomous vehicle does not have the right of way, the pedestrian tracking system is further adapted to con

Assignees

• Gm Global Tech Operations Llc

Inventors

• Khamis Alaa M
• Tong Wei

Classifications

• B60W2554/4029

  Pedestrians · CPC title

• B60W2556/45

  External transmission of data to or from the vehicle · CPC title

• B60W2420/54

  Audio sensitive means, e.g. ultrasound · CPC title

• B60W2552/53

  Road markings, e.g. lane marker or crosswalk · CPC title

• B60W2554/4046

  Behavior, e.g. aggressive or erratic · CPC title