Intelligent vehicle systems and control logic for incident prediction and assistance in off-road driving situations

Patent metadata

Field	Value
Publication number	US-12014552-B2
Application number	US-202117544195-A
Country	US
Kind code	B2
Filing date	Dec 7, 2021
Priority date	Dec 7, 2021
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

Presented are intelligent vehicle systems for off-road driving incident prediction and assistance, methods for making/operating such systems, and vehicles networking with such systems. A method for operating a motor vehicle includes a system controller receiving geolocation data indicating the vehicle is in or entering off-road terrain. Responsive to the vehicle geolocation data, the controller receives, from vehicle-mounted cameras, camera-generated images each containing the vehicle's drive wheel(s) and/or the off-road terrain's surface. The controller receives, from a controller area network bus, vehicle operating characteristics data and vehicle dynamics data for the motor vehicle. The camera data, vehicle operating characteristics data, and vehicle dynamics data is processed via a convolutional neural network backbone to predict occurrence of a driving incident on the off-road terrain within a prediction time horizon. The system controller commands a resident vehicle system to execute a control operation responsive to the predicted occurrence of the driving incident.

First claim

What is claimed:

1. A method for controlling operation of a motor vehicle with a sensor array including a network of cameras mounted at discrete locations on the motor vehicle, the method comprising:
receiving, via a system controller through a wireless communications device, vehicle geolocation data indicating the motor vehicle is in or entering an off-road terrain;
receiving, via the system controller from the sensor array responsive to the vehicle geolocation data, camera data indicative of camera-generated images captured by the cameras and containing a drive wheel of the motor vehicle and a terrain surface of the off-road terrain;
receiving, via the system controller from a controller area network (CAN) bus, vehicle operating characteristics data and vehicle dynamics data for the motor vehicle;
processing the camera data, the vehicle operating characteristics data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict occurrence of a driving incident on the off-road terrain within a prediction time horizon; and
transmitting, via the system controller, a command signal to a resident vehicle system to execute a control operation responsive to the predicted occurrence of the driving incident.

2. The method of claim 1 , further comprising:
determining a region of interest (ROI) inset within and fixed at a predefined location of a camera view of each of the cameras; and
generating cropped images by cropping each of the camera-generated images to remove image data outside of the ROI.

3. The method of claim 2 , wherein processing the camera data includes the CNN backbone analyzing the cropped images to determine a wheel characteristic for the drive wheel of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the wheel characteristic for the drive wheel.

4. The method of claim 3 , wherein the wheel characteristic includes a loss of ground contact status, a suspended in impediment status, and/or a loss of normal operating feature status.

5. The method of claim 2 , wherein processing the camera data includes the CNN backbone analyzing the cropped images to determine a terrain characteristic for the terrain surface of the off-road terrain, wherein predicting the occurrence of the driving incident is based on the terrain characteristic for the terrain surface.

6. The method of claim 5 , wherein the terrain characteristic includes a terrain type and/or a terrain condition.

7. The method of claim 2 , wherein processing the camera data includes the CNN backbone analyzing the cropped images to determine an obstacle characteristic for an obstacle obstructing driving of the motor vehicle, wherein predicting the occurrence of the driving incident is based on the obstacle characteristic for the obstacle.

8. The method of claim 7 , wherein the obstacle characteristic includes an obstacle height relative to a wheel height of the drive wheel and/or a body height of a vehicle body of the motor vehicle.

9. The method of claim 2 , wherein processing the camera data includes:
analyzing, for each of the camera-generated images independently from one another, a single frame of the ROI to assign a characteristic to the drive wheel and/or the terrain surface contained in the camera-generated image; and
generating, via the CNN backbone using multiple task heads based on the analysis of the single frame of the ROI, a situational cue indicative of the characteristic and evaluable via a situation classifier module operable to predict the occurrence of the driving incident.

10. The method of claim 2 , wherein processing the camera data includes:
analyzing, via the CNN backbone for all of the camera-generated images, consecutive frames of the ROI to extract features of the drive wheel and/or the terrain surface contained in the camera-generated image;
linking, via a concatenation module, the camera-generated images in a series or chain; and
extracting, via a recursive neural network (RNN), temporal information for each of the camera-generated images.

11. The method of claim 1 , wherein the network of cameras includes an underbody camera mounted to a vehicle body of the motor vehicle proximate an undercarriage thereof, the underbody camera being operable to capture an outboard-facing downward view from the vehicle body and generate signals indicative thereof.

12. The method of claim 1 , wherein the vehicle dynamics data includes vehicle roll data, vehicle pitch data, vehicle yaw data, vehicle lateral/longitudinal speed data, vehicle lateral/longitudinal acceleration data, wheel speed data, and/or steering angle data, and wherein the vehicle operating characteristics data includes throttle position data, brake force data, parking brake status, powertrain mode data, and/or suspension height data.

13. The method of claim 1 , wherein the resident vehicle system includes an autonomous driving control module operable to automate driving of the motor vehicle, the control operation including automating a driving maneuver of the motor vehicle based on the predicted occurrence of the driving incident.

14. The method of claim 1 , wherein the resident vehicle system includes a vehicle navigation system with a display device, the control operation including displaying, via the display device, an alert with a remediating driving maneuver to prevent occurrence of the driving incident.

15. A non-transitory, computer-readable medium storing instructions executable by a system controller operable to control operation of a motor vehicle, the motor vehicle including a sensor array with a network of cameras mounted at discrete locations on the motor vehicle, the instructions, when executed, causing the system controller to perform operations comprising:
receiving, via a wireless communications device, vehicle geolocation data indicating the motor vehicle is in or entering an off-road terrain;
receiving, from the sensor array, camera data indicative of camera-generated images captured by the cameras and each containing a drive wheel of the motor vehicle and/or a terrain surface of the off-road terrain;
receiving, from a controller area network (CAN) bus of the motor vehicle, vehicle operating characteristics data and vehicle dynamics data for the motor vehicle;
processing the camera data, the vehicle operating characteristics data, and the vehicle dynamics data via a convolutional neural network (CNN) backbone to predict occurrence of a driving incident on the off-road terrain within a prediction time horizon; and
transmitting a command signal to a resident vehicle system to execute a control operation responsive to the predicted occurrence of the driving incident.

16. A motor vehicle, comprising:
a vehicle body;
multiple drive wheels mounted to the vehicle body;
a prime mover mounted to the vehicle body and operable to drive one or more of the drive wheels to thereby propel the motor vehicle;
a sensor array including a network of cameras mounted at discrete locations on the vehicle body; and
an electronic system controller programmed to:
receive, through a wireless communications device, vehicle geolocation data indicating the motor vehicle is in or entering an off-road terrain;
receive, from the sensor array responsive to the vehicle geolocation data, camera data indicative of camera-generated images captured by the cameras and each containing one or more of the drive wheels and/or a terrain surface of the off-road terrain;
receive, from a controller area network (CAN) bus, vehicle operating characteristics data and vehicle dynamics data for the motor veh

Assignees

• Gm Global Tech Operations Llc

Inventors

• Telpaz Ariel
• Erez Ravid
• Baltaxe Michael
• Hershkovitz Barak
• Baron Nadav
• Stanek Christopher A

Classifications

• G05D1/617

  Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80) · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06V10/62

  relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking · CPC title

• G07C5/02

  Registering or indicating driving, working, idle, or waiting time only (apparatus forming part of taximeters G07B13/00) · CPC title