Reinforcement learning control of vehicle systems

Patent metadata

Field	Value
Publication number	US-12195020-B2
Application number	US-202016906674-A
Country	US
Kind code	B2
Filing date	Jun 19, 2020
Priority date	Jun 20, 2019
Publication date	Jan 14, 2025
Grant date	Jan 14, 2025

Abstract

A system includes a sensor array and a processing circuit. The processing circuit is operable to: store a policy; receive the sensor information from the sensor array; receive horizon information from a horizon system; input the sensor information and the horizon information into the policy; determine an output of the policy based on the input of the sensor information and the horizon information; control operation of a vehicle system according to the output; compare the sensor information received after controlling operation of the vehicle system according to the output relative to a reward or penalty condition; provide one of a reward signal or a penalty signal in response to the comparison; update the policy based on receipt of the reward signal or the penalty signal; and control the vehicle system using the updated policy to improve operation in view of the operating parameter.

First claim

What is claimed is:

1. A system, comprising:
a sensor array structured to provide sensor information indicative of an operating parameter regarding operation of a diesel exhaust fluid doser of a vehicle, the diesel exhaust fluid doser structured to introduce an aqueous urea solution for use by a selective catalyst reduction (SCR) catalyst to reduce NOx in an exhaust gas stream; and
a processing circuit coupled to the sensor array, the processing circuit comprising one or more memory devices coupled to one or more processors, the one or more memory devices configured to store instructions thereon that, when executed by the one or more processors, cause the one or more processors to:
store, in the one or more memory devices, a machine learning model for controlling operation of the diesel exhaust fluid doser of the vehicle;
receive the sensor information regarding the operation of the diesel exhaust fluid doser of the vehicle from the sensor array;
receive horizon information indicative of at least one upcoming road condition;
input the sensor information and the horizon information into the machine learning model;
determine an output of the machine learning model based on the input of the sensor information and the horizon information, the output of the machine learning model indicative of a control action for controlling the diesel exhaust fluid doser of the vehicle;
control the operation of the diesel exhaust fluid doser of the vehicle according to the control action indicated by the output of the machine learning model;
compare the sensor information received after controlling the operation of the diesel exhaust fluid doser of the vehicle according to the control action indicated by the output of the machine learning model to a desired value of the operating parameter regarding a reward or penalty condition;
provide one of a reward signal or a penalty signal in response to the comparison;
update the machine learning model based on receipt of the reward signal or the penalty signal; and
control the diesel exhaust fluid doser of the vehicle using the updated machine learning model to improve operation in view of the operating parameter.

2. The system of claim 1 , wherein the one or more memory devices are located in the vehicle.

3. The system of claim 1 , wherein the horizon information includes look ahead information including at least one of an altitude, a grade, or a turn degree information.

4. The system of claim 1 , wherein the instructions, when executed by the one or more processors, cause the one or more processors to receive fleet information from other vehicles, and utilize the fleet information to update the machine learning model.

5. The system of claim 1 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to determine an age of an engine component and update the machine learning model using the age of the engine component.

6. A system comprising:
a diesel exhaust fluid doser of a vehicle structured to introduce an aqueous urea solution for use by a selective catalyst reduction (SCR) catalyst to reduce NOx in an exhaust gas stream;
a sensor array coupled to the diesel exhaust fluid doser of the vehicle, the sensor array structured to provide sensor information of an operating parameter regarding operation of the diesel exhaust fluid doser of the vehicle; and
a processing circuit comprising one or more memory devices coupled to one or more processors, the one or more memory devices configured to store instructions thereon that, when executed by the one or more processors, cause the one or more processors to:
store, in the one or more memory devices, a machine learning model for controlling the operation of the diesel exhaust fluid doser of the vehicle;
receive the sensor information regarding the operation of the diesel exhaust fluid doser of the vehicle from the sensor array;
receive age information indicating an operational age of the diesel exhaust fluid doser of the vehicle;
input the sensor information and the age information into the machine learning model;
determine an output of the machine learning model based on the input of the sensor information and the age information, the output of the machine learning model indicative of a control action for controlling the diesel exhaust fluid doser of the vehicle;
control the operation of the diesel exhaust fluid doser of the vehicle according to the control action indicated by the output of the machine learning model;
compare the sensor information received after controlling the operation of the diesel exhaust fluid doser of the vehicle according to the control action indicated by the output of the machine learning model to a desired value of the operating parameter regarding a reward or penalty condition;
provide one of a reward signal or a penalty signal in response to the comparison; and
update the machine learning model based on receipt of the reward signal or the penalty signal.

7. The system of claim 6 , wherein the age information includes a historical age based on performance information.

8. The system of claim 6 , wherein the age information is saved to the one or more memory devices and provided to a second vehicle system associated with a different vehicle.

9. The system of claim 6 , wherein the age information is updated in real time by the one or more processors.

10. The system of claim 6 , wherein the one or more memory devices are located within the vehicle.

11. The system of claim 6 , wherein the age information is received from a look up table, and wherein the look up table is queried using the sensor information by the one or more processors.

12. The system of claim 6 , wherein the sensor information includes horizon information indicative of upcoming roadway conditions.

Assignees

• Cummins Inc

Inventors

• Borhan Hoseinali
• Salemian Abbas Hooshmand
• Hodzen Edmund P

Classifications

• F01N3/208

  Control of selective catalytic reduction [SCR], e.g. by adjusting the dosing of reducing agent · CPC title

• F02D41/0002

  Controlling intake air · CPC title

• F01N2610/02

  the substance being ammonia or urea · CPC title

• B60W30/14

  {Adaptive} cruise control · CPC title

• F02D33/00

  Controlling delivery of fuel or combustion-air, not otherwise provided for {(using exhaust gas sensors F02D35/0023, F02D35/0046)} · CPC title