Free space detection using rgb-polarimetric images

Patent metadata

Field	Value
Publication number	US-2024257527-A1
Application number	US-202318161211-A
Country	US
Kind code	A1
Filing date	Jan 30, 2023
Priority date	Jan 30, 2023
Publication date	Aug 1, 2024
Grant date	—

Abstract

A free space estimation and visualization system for a host vehicle includes a camera configured to collect red-green-blue (“RGB”)-polarimetric image data of drive environs of the host vehicle, including a potential driving path. An electronic control unit (“ECU”) receives the RGB-polarimetric image data and estimates free space in the driving path by processing the RGB-polarimetric image data via a run-time neural network. Control actions are taken in response to the estimated free space. A method for use with the visualization system includes collecting RGB and lidar data of target drive scenes and generating, via a first neural network, pseudo-labels of the scenes. The method includes collecting RGB-polarimetric data via a camera and thereafter training a second neural network using the RGB-polarimetric data and pseudo-labels. The second neural network is used in the ECU to estimate free space in the potential driving path.

First claim

What is claimed is:

1 . A free space estimation and visualization system for a host vehicle, comprising:
a camera configured to collect red-green-blue (“RGB”)-polarimetric image data of drive environs of the host vehicle, including a potential driving path of the host vehicle; and an electronic control unit (“ECU”) in communication with the camera and configured to:
receive the RGB-polarimetric image data from the camera;
estimate an amount of free space in the potential driving path as estimated free space, including processing the RGB-polarimetric image data via a run-time neural network; and
execute a control action aboard the host vehicle in response to the estimated free space.

2 . The system of claim 1 , wherein the ECU is configured to calculate a feature set using the RGB-polarimetric image data, and to communicate the feature set to the run-time neural network as an input data set, the input data set being characterized by an absence of lidar data.

3 . The system of claim 2 , wherein the feature set has six set elements determined as a concatenation of RGB data, AoLP data, and DoLP data from the camera.

4 . The system of claim 3 , wherein the six set elements include sin(2·AoLP), cos(2·AoLP), 2·DolP−1, 2·R−1, 2·G−1, and 2·B−1.

5 . The system of claim 1 , wherein the host vehicle is a motor vehicle having a vehicle body, and wherein the camera is connected to the vehicle body.

6 . The system of claim 1 , wherein the ECU is in communication with a path planning module of the host vehicle, and is configured to provide the estimated free space to the path planning module as at least part of the control action.

7 . The system of claim 1 , wherein the ECU is in communication with a display screen and configured to display a graphical representation of the estimated free space on the display screen.

8 . A method for use with a free space estimation and visualization system, comprising:
collecting red-green-blue (“RGB”) data and lidar data of a target drive scene using an RGB camera and a lidar sensor, respectively; generating, via a first neural network of a training computer, pseudo-labels of the target drive scene; collecting RGB-polarimetric data via an RGB-polarimeric camera; training a second neural network of the training computer using the RGB-polarimetric data and the pseudo-labels; using the second neural network in an electronic control unit (“ECU”) of the host vehicle as a run-time neural network to estimate an amount of free space in a potential driving path of the host vehicle as estimated free space, including processing additional RGB-polarimetric image data via the run-time neural network; and executing a control action aboard the host vehicle in response to the estimated free space.

9 . The method of claim 8 , further comprising:
calculating a feature set using the RGB-polarimetric image data; and communicating the feature set to the run-time neural network as an input data set, wherein the input data set is characterized by an absence of lidar data.

10 . The method of claim 9 , wherein calculating the feature set using the RGB-polarimetric image data includes calculating the feature set with six set elements as a concatenation of RGB data, AoLP data, and DoLP data from the camera.

11 . The method of claim 10 , wherein calculating the feature set with six set elements includes calculating sin(2·AoLP), cos(2·AoLP), 2·DolP−1, 2·R−1, 2 G−1, and 2·B−1 as the six claim elements.

12 . The method of claim 9 , wherein the host vehicle is a motor vehicle having a vehicle body, the camera is a body-mounted RGB-polarimetric camera connected to the vehicle body, and collecting the RGB-polarimetric data via the body-mounted RGB-polarimetric camera.

13 . The method of claim 9 , wherein the ECU is in communication with a path planning control module of the host vehicle, the method further comprising:
planning a drive path of the host vehicle as at least part of the control action.

14 . The method of claim 9 , wherein the ECU is in communication with a display screen, the method further comprising:
displaying a graphical representation of the estimated free space on the display screen.

15 . A host vehicle, comprising:
a vehicle body; road wheels connected to the vehicle body; and a free space estimation and visualization (“FSEV”) system including:
a camera connected to the vehicle body and configured to collect red-green-blue (“RGB”)-polarimetric image data of drive environs of the host vehicle, including a potential driving path thereof, and
an electronic control unit (“ECU”) in communication with the camera and configured to:
receive the RGB-polarimetric image data from the camera;
estimate an amount of free space in the potential driving path as estimated free space, including processing the RGB-polarimetric image data via a run-time neural network; and
execute a control action aboard the host vehicle in response to the estimated free space.

16 . The host vehicle of claim 15 , wherein the ECU is configured to calculate a feature set using the RGB-polarimetric image data, and to communicate the feature set to the run-time neural network as an input data set, the input data set being characterized by an absence of lidar data.

17 . The host vehicle of claim 16 , wherein the feature set has six set elements determined as a concatenation of RGB data, AoLP data, and DoLP data from the camera.

18 . The host vehicle of claim 17 , wherein the six set elements include sin(2·AoLP), cos(2·AoLP), 2·DolP−1, 2·R−1, 2·G−1, and 2·B−1.

19 . The host vehicle of claim 15 , further comprising:
a path planning module in communication with the ECU, wherein the ECU is configured to communicate the estimated free space to the path planning module as part of the control action, and wherein the path planning module is configured to plan a drive path of the host vehicle in response to the estimated free space.

20 . The host vehicle of claim 15 , further comprising:
a display screen in communication with the ECU, wherein the ECU is configured to display a graphical representation of the estimated free space on the display screen as at least part of the control action.

Assignees

• Gm Global Tech Operations Llc

Inventors

• Baltaxe Michael
• Pe'Er Tomer
• Levi Dan

Classifications

• B60W2050/0043

  Signal treatments, identification of variables or parameters, parameter estimation or state estimation · CPC title

• B60W2050/0001

  Details of the control system · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T7/90

  Determination of colour characteristics · CPC title