Vehicle personal assistant that interprets spoken natural language input based upon vehicle context

Patent metadata

Field	Value
Publication number	US-9798799-B2
Application number	US-201213678209-A
Country	US
Kind code	B2
Filing date	Nov 15, 2012
Priority date	Nov 15, 2012
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

A vehicle personal assistant to engage a user in a conversational dialog about vehicle-related topics, such as those commonly found in a vehicle owner's manual, includes modules to interpret spoken natural language input, search a vehicle knowledge base and/or other data sources for pertinent information, and respond to the user's input in a conversational fashion. The dialog may be initiated by the user or more proactively by the vehicle personal assistant based on events that may be currently happening in relation to the vehicle. The vehicle personal assistant may use real-time inputs obtained from the vehicle and/or non-verbal inputs from the user to enhance its understanding of the dialog and assist the user in a variety of ways.

First claim

The invention claimed is:

1. A computer-implemented method, comprising:
receiving, by a vehicle computing system, natural language input interpreted from audio input, wherein the natural language input is associated with a vehicle, and wherein the natural language input is received after an interval of time has passed since a previous input was received;
receiving visual input, wherein the visual input is associated with the vehicle, and wherein the visual input includes an image of a person when the person is in proximity to the vehicle;
determining a current context of the vehicle, wherein determining includes using vehicle-related inputs;
determining that the natural language input is associated with a previous dialog, wherein the previous dialog includes the previous input, wherein determining includes using a vehicle-specific conversation model, wherein the vehicle-specific conversation model includes one or more rules for determining when a conversation has started or ended, and wherein the vehicle-specific conversation model associates the interval of time with a pause that is associated with an operation related to the vehicle;
determining a query, wherein the query is determined using the natural language input, the visual input, the previous dialog, and the current context of the vehicle;
using the query to determine information about the vehicle, wherein the information is determined using a vehicle user guide; and
presenting the information using a component of the vehicle.

2. The computer-implemented method of claim 1 , wherein the information about the vehicle includes information for troubleshooting a component of the vehicle.

3. The computer-implemented method of claim 1 , wherein determining that the natural language input is associated with the previous dialog includes determining that the previous input was incomplete.

4. The computer-implemented method of claim 1 , wherein determining that the natural language input is associated with the previous dialog includes determining content associated with the natural language input and determining that the previous dialog includes the associated content.

5. The computer-implemented method of claim 1 , wherein the vehicle-specific conversation model incudes one or more rules for determining whether the natural language input is associated with other inputs, and wherein the other inputs include the vehicle-related inputs.

6. The computer-implemented method of claim 1 , wherein determining the query includes using touch input generated by one or more touch sensors in the vehicle.

7. The computer-implemented method of claim 1 , wherein the current context of the vehicle includes a current operational state of a component the vehicle.

8. The computer-implemented method of claim 1 , wherein determining the current context of the vehicle includes using historical information to determine a particular driving situation for the vehicle, wherein a driving situation describes a behavior related to the vehicle, and wherein the historical information records one or more recent operational states of the vehicle.

9. The computer-implemented method of claim 1 , further comprising:
using the visual input to determine a geospatial reference, wherein a geospatial reference identifies a point in space related to the vehicle.

10. The computer-implemented method of claim 1 , further comprising:
determining that additional information is needed to determine the query; and
executing an action to solicit the additional information.

11. The computer-implemented method of claim 1 , further comprising:
modifying the query, wherein the query is modified using the current context of the vehicle.

12. The computer-implemented method of claim 1 , wherein the information about the vehicle is determined using an external data source, wherein the external data source is accessible using a network interface of the vehicle.

13. A vehicle computing system, comprising:
one or more processors; and
a non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
receiving natural language input interpreted from audio input, wherein the natural language input is associated with a vehicle, and wherein the natural language input is received after an interval of time has passed since a previous input was received;
receiving visual input, wherein the visual input is associated with the vehicle, and wherein the visual input includes an image of a person when the person is in proximity to the vehicle;
determining a current context of the vehicle, wherein determining includes using vehicle-related inputs;
determining that the natural language input is associated with a previous dialog, wherein the previous dialog includes the previous input, wherein determining includes using a vehicle-specific conversation model, wherein the vehicle-specific conversation model includes one or more rules for determining when a conversation has started or ended, and wherein the vehicle-specific conversation model associates the interval of time with a pause that is associated with an operation related to the vehicle;
determining a query, wherein the query is determined using the natural language input, the visual input, the previous dialog, and the current context of the vehicle;
using the query to determine information about the vehicle, wherein the information is determined using a vehicle user guide; and
presenting the information using a component of the vehicle.

14. The vehicle computing system of claim 13 , wherein the information about the vehicle includes information for troubleshooting a component of the vehicle.

15. The vehicle computing system of claim 13 , wherein the instructions for determining that the natural language input is associated with the previous dialog include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
determining that the previous input was incomplete.

16. The vehicle computing system of claim 13 , wherein the instructions for determining that the natural language input is associated with the previous dialog include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
determining content associated with the natural language input and determining that the previous dialog includes the associated content.

17. The vehicle computing system of claim 13 , wherein the vehicle-specific conversation model includes one or more rules for determining whether the natural language input is associated with other inputs, and wherein the other inputs include the vehicle-related inputs.

18. The vehicle computing system of claim 13 , wherein the instructions for determining the query include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
using touch input generated by one or more touch sensors in the vehicle.

19. The vehicle computing system of claim 13 , wherein the current context of the vehicle includes a current operational state of a component the vehicle.

20. The vehicle computing system of claim 13 , wherein the instructions for determining the current context of the vehicle include instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
using histor

Assignees

• Stanford Res Inst Int

Inventors

• Wolverton Michael J
• Mark William S
• Bratt Harry
• Bercow Douglas A

Classifications

• G10L2015/228

  of application context · CPC title

• G06F16/3329 Primary

  Natural language query formulation · CPC title

• G10L15/22

  Procedures used during a speech recognition process, e.g. man-machine dialogue · CPC title

• G10L2015/227

  of the speaker; Human-factor methodology · CPC title

• G10L2015/226

  using non-speech characteristics · CPC title