User-assisted maintenance of autonomous vehicle fleet

Patent metadata

Field	Value
Publication number	US-11505202-B2
Application number	US-202016907505-A
Country	US
Kind code	B2
Filing date	Jun 22, 2020
Priority date	Jun 22, 2020
Publication date	Nov 22, 2022
Grant date	Nov 22, 2022

Abstract

An autonomous vehicle (AV) maintenance system engages a user of an AV to assist in maintenance tasks. A sensor interface receives data captured by a sensor of the AV. An issue detector processes the sensor data to detect an issue in the AV. A user interface module instructs a user to perform a maintenance task addressing the detected issue. A verification module verifies that the user successfully completed the maintenance task.

First claim

What is claimed is:

1. A system for autonomous vehicle (AV) maintenance comprising:
a sensor interface configured to receive data captured by a sensor, the data describing a state of a component of the AV;
an issue detector configured to detect an issue related to the component of the AV based on the data captured by the sensor;
a vehicle maintenance controller configured to:
retrieve user data describing a user of the AV, the user data based on a response of the user to a previous request to the user to address a previous issue; and
determine, based on the retrieved user data and the detected issue, to surface an instruction to the user to address the detected issue;

a user interface (UI) module configured to generate the instruction for the user of the AV for presentation via a user interface; and
a verification module configured to verify, based on additional data captured by the sensor and describing an updated state of the component of the AV, that the issue related to the component of the AV has been resolved.

2. The system of claim 1 , wherein the issue detector is further configured to:
retrieve a machine-learned model for detecting the issue, the machine-learned model trained to detect the issue related to the component based on data captured by a plurality of sensors similar to the sensor; and
determine, using the machine-learned model, that the component of the AV is exhibiting the issue.

3. The system of claim 2 , wherein the sensor is a camera configured to capture an image of a portion of the AV, and the machine-learned model is configured to determine whether the portion of the AV is in its expected state or an unexpected state.

4. The system of claim 2 , wherein the verification module is configured to use the machine-learned model to determine that the additional data captured by the sensor does not indicate an issue with the component of the AV.

5. The system of claim 1 , wherein the sensor is a sensor configured to monitor an environment of the AV and capture data used by the AV for autonomous driving, the detected issue is an obscuring of at least a portion of a field of view of the sensor, and the instruction for the user of the AV comprises an instruction to remove material obscuring the sensor from the sensor.

6. The system of claim 5 , wherein the UI module further configured to:
generate, in the instruction for the user, a visual representation of the field of view of the sensor, the visual representation showing the obscuring of the at least a portion of the field of view; and
generate a second instruction for the user, the second instruction comprising an updated visual representation of the field of view of the sensor, the updated visual representation based on the additional data captured by the sensor, and the second instruction further comprising a request for the user to indicate whether the field of view is no longer obscured.

7. The system of claim 5 , wherein the UI module is further configured to generate a second instruction for the user to move to a position within the field of view of the sensor, and the verification module is further configured to verify that the user is visible to the sensor in the field of view.

8. A method for autonomous vehicle (AV) maintenance comprising:
receiving data captured by a sensor, the data describing a state of a component of the AV;
detecting an issue related to the component of the AV based on the data captured by the sensor;
retrieving user data describing a user of the AV, the user data based on a response of the user to a previous request to the user to address a previous issue;
determining, based on the retrieved user data and the detected issue, to surface an instruction to the user to address the detected issue;
generating the instruction for the user of the AV for presentation via a user interface;
receiving additional data captured by the sensor, the additional data describing an updated state of the component of the AV; and
verifying, based on the received additional data, that the issue related to the component of the AV has been resolved.

9. The method of claim 8 , wherein detecting an issue related to the component of the AV based on the data captured by the sensor comprises:
retrieving a machine-learned model for detecting the issue, the machine-learned model trained to identify the issue related to the component based on data captured by a plurality of sensors similar to the sensor; and
determining, using the machine-learned model, that the component of the AV is exhibiting the issue.

10. The method of claim 9 , wherein the sensor is a camera configured to capture an image of a portion of the AV, and the machine-learned model determines whether the portion of the AV is in its expected state or an unexpected state.

11. The method of claim 9 , wherein verifying that the issue related to the component of the AV has been resolved comprises determining, using the machine-learned model, that the additional data captured by the sensor does not indicate an issue with the component of the AV.

12. The method of claim 8 , wherein the sensor is a sensor configured to monitor an environment of the AV and capture data used by the AV for autonomous driving, the detected issue is an obscuring of at least a portion of a field of view of the sensor, and the instruction for the user of the AV comprises an instruction to remove material obscuring the sensor from the sensor.

13. The method of claim 12 , wherein the instruction for the user comprises a visual representation of the field of view of the sensor, the visual representation showing the obscuring of the at least a portion of the field of view, the method further comprising:
generating an updated visual representation of the field of view of the sensor, the updated visual representation based on the additional data captured by the sensor; and
receiving from the user, via the user interface, an indication that the field of view is no longer obscured.

14. The method of claim 12 , further comprising generating a second instruction for the user to move to a position within the field of view of the sensor, wherein verifying that the issue related to the component of the AV has been resolved comprises verifying that the user is visible to the sensor in the field of view.

15. The method of claim 8 , wherein the user data indicates whether the user completed a previous a maintenance task related to the previous request.

16. The method of claim 8 , wherein the user data indicates whether the user agreed to perform a maintenance task related to the previous request.

17. A system for maintaining a fleet of autonomous vehicles (AVs), the system comprising:
a fleet maintenance manager configured to:
receive, from an AV of the AV fleet, data describing a maintenance issue detected by the AV based on sensor data;
receive, from the AV, a confirmation that a user of the AV attended to and corrected the detected maintenance issue;

a vehicle maintenance controller configured to:
retrieve user data describing the user of the AV, the user data based on a response of the user to a previous request to the user to address a previous maintenance issue; and
determine, based on the retrieved user data and the detected issue, to surface an instruction to the user to address the detected issue;

a user interface module configured to transmit, to a user device, the instruction for the user to correct the detected maintenance issue; and
a user database configured to store data associating users of the AV fleet with maintenance issues addressed by

Assignees

• Gm Cruise Holdings Llc

Inventors

• Zhang Yifei
• Shaw Jeffrey
• Gerrese Alexander Willem
• Juel Jeremy Stephen

Classifications

• G05B23/0283

  Predictive maintenance, e.g. involving the monitoring of a system and, based on the monitoring results, taking decisions on the maintenance schedule of the monitored system; Estimating remaining useful life [RUL] (preventive maintenance, i.e. planning maintenance according to the available resources without monitoring the system G06Q10/06) · CPC title

• G05B2219/2637

  Vehicle, car, auto, wheelchair · CPC title

• B60W2050/146

  Display means · CPC title

• B60W50/04

  Monitoring the functioning of the control system · CPC title

• G06Q10/20 Primary

  Administration of product repair or maintenance · CPC title