Shared checklists for onboard assistant

Patent metadata

Field	Value
Publication number	US-12597040-B2
Application number	US-202217869460-A
Country	US
Kind code	B2
Filing date	Jul 20, 2022
Priority date	Jul 20, 2022
Publication date	Apr 7, 2026
Grant date	Apr 7, 2026

Abstract

There is disclosed herein a method of providing customer service, including receiving a customer service request from an autonomous vehicle (AV), the customer service request comprising sensor data collected by the AV; selecting a standard operating procedure (SOP) for responding to the customer service request, the SOP selected from a data store comprising a plurality of SOPs, wherein selecting the SOP is based at least in part on the sensor data from the AV, and wherein the SOP includes a human-usable resolution script; and handling the customer service request, including displaying the human-usable resolution script to a human customer service agent to assist the human customer service agent in handling the customer service request.

First claim

What is claimed is:

1 . A method of providing customer service, comprising:
receiving, by an electronic autonomous vehicle (AV) operator, a customer service (CS) request from an AV, the CS request comprising first sensor data collected by the AV; mapping, by the electronic AV operator, the CS request to a cause, and a first standard operating procedure (SOP) for responding to the CS request, wherein the first SOP includes a first human-usable resolution script wherein mapping the first SOP comprises accounting for a user sentiment of a passenger of the AV based on the first sensor data, including camera data and microphone data, wherein mapping the first SOP further comprises providing the first sensor data to a machine learning (ML) algorithm, and selecting the first SOP as an output of the ML algorithm; displaying, by the electronic AV operator, the first human-usable resolution script and cause to a human CS agent to assist the human CS agent in handling the CS request after displaying the first human-usable resolution script, receiving, by the electronic AV operator, second sensor data collected by the AV subsequent to the first sensor data; based on the second sensor data, determining, by the electronic AV operator, a change in condition of the AV, the change in condition comprising at least one of a change in AV operational state or a change in the user sentiment; based on determining the change in condition of the AV, selecting, by the electronic AV operator using the ML algorithm, a second SOP comprising a data structure linking a second human-usable resolution script to a set of computer-executable instructions corresponding to the determined change in condition; and updating, by the electronic AV operator, the display for the human CS agent to replace the first human-usable resolution script with the second human-usable resolution script corresponding to the second SOP; and transmitting, by the electronic AV operator to the AV, a command derived from the set of machine instructions of the second SOP to modify an operational state of the AV relevant to the determined change in condition, wherein modifying the operational state comprises one of:
dispatching a replacement AV to a location of the AV;
dispatching a support vehicle to the location of the AV;
or rerouting the AV to avoid a hazard.

2 . The method of claim 1 , further comprising training the ML algorithm with a plurality of real and simulated sensor data conditions and respective preferred SOPs for the real and simulated sensor data conditions.

3 . The method of claim 2 , wherein the ML algorithm is a deep learning (DL) neural network.

4 . The method of claim 1 , wherein the change in condition of the AV is based on thermal imaging data.

5 . The method of claim 4 , wherein the second human-usable resolution script comprises suggested phraseology tailored to address the user sentiment inferred from the thermal imaging data.

6 . The method of claim 4 , wherein the thermal imaging data indicates a passenger body temperature metric crossing a predefined threshold.

7 . The method of claim 4 , wherein the change in condition comprises a change in AV operational state.

8 . The method of claim 4 , wherein the change in condition comprises a change in the user sentiment.

9 . The method of claim 1 , wherein the ML algorithm is trained on sets of sensor inputs along with a correct SOP selected for each set of sensor inputs,
wherein the sets of sensor inputs comprise at least one set of real world sensor data and at least one set of simulated sensor data.

10 . The method of claim 1 , wherein the first sensor data includes infrared data for use in accounting for the user sentiment of the passenger for selecting the first SOP.

11 . The method of claim 1 , wherein the first SOP includes a set of steps for achieving a set of goals.

12 . A computing system comprising:
at least one processor circuit, and at least one memory; and instructions encoded on the at least one memory that, when executed, instruct the at least one processor circuit to:
receive a customer service request comprising first sensor data collected by an autonomous vehicle (AV);
map the CS request to a cause, and a first standard operating procedure (SOP) for responding to the CS request, wherein the first SOP includes a first human-usable resolution script wherein mapping the first SOP comprises accounting for a user sentiment of a passenger of the AV based on the first sensor data, including camera data and microphone data, wherein mapping the first SOP further comprises providing the first sensor data to a machine learning (ML) algorithm, and selecting the first SOP as an output of the ML algorithm;
display the first human-usable resolution script and cause to a human CS agent to assist the human CS agent in handling the CS request
after displaying the first human-usable resolution script, receive second sensor data collected by the AV subsequent to the first sensor data;
based on the second sensor data, determine a change in condition of the AV, the change in condition comprising at least one of a change in AV operational state or a change in the user sentiment;
based on determining the change in condition of the AV, select, using the ML algorithm, a second SOP comprising a data structure linking a second human-usable resolution script to a set of computer-executable instructions corresponding to the determined change in condition; and
update the display for the human CS agent to replace the first human-usable resolution script with the second human-usable resolution script corresponding to the second SOP; and
transmit a command derived from the set of machine instructions of the second SOP to modify an operational state of the AV relevant to the determined change in condition, wherein modifying the operational state comprises one of:
dispatching a replacement AV to a location of the AV;
dispatching a support vehicle to the location of the AV;
or rerouting the AV to avoid a hazard.

13 . The computing system of claim 12 , wherein the instructions are further to train the ML algorithm with a plurality of real and simulated sensor data conditions and respective preferred SOPs for the real and simulated sensor data conditions.

14 . The computing system of claim 13 , wherein the ML algorithm is a deep learning (DL) neural network.

15 . The computing system of claim 12 , wherein the change in condition of the AV is based on thermal imaging data.

16 . The computing system of claim 15 , wherein the second human-usable resolution script comprises suggested phraseology tailored to address the user sentiment inferred from the thermal imaging data.

17 . The computing system of claim 15 , wherein the thermal imaging data indicates a passenger body temperature metric crossing a predefined threshold.

18 . One or more tangible, non-transitory computer-readable media having stored thereon executable instructions to instruct a processor to:
receive a customer service request comprising first sensor data collected by an autonomous vehicle (AV); map the CS request to a cause, and a first standard operating procedure (SOP) for responding to the CS request, wherein the first SOP includes a first human-usable resolution script wherein mapping the first SOP comprises accounting for a user sentiment of a passenger of the AV based on the first sensor data, including camera data and microphone data, wherein mapping the first SOP further comprises providing the first sensor data to a machine learni

Assignees

• Gm Cruise Holdings Llc

Inventors

• Warren Christopher
• Tong Minjie
• Foy Theresa
• Sams Ashley
• Phan Soleil
• Meyerholtz Leah
• Shuckerow David J
• Thengvall Daniel Paul
• Li Qingxuan
• Sturkop Steven

Classifications

• G06Q30/015

  Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk · CPC title

• B60W40/08

  related to drivers or passengers · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• B60W2540/01

  Occupants other than the driver · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title