Gesture based control of autonomous vehicles

Patent metadata

Field	Value
Publication number	US-10913463-B2
Application number	US-201715709389-A
Country	US
Kind code	B2
Filing date	Sep 19, 2017
Priority date	Sep 21, 2016
Publication date	Feb 9, 2021
Grant date	Feb 9, 2021

Abstract

A triggering condition for initiating an interaction session with an occupant of a vehicle is detected. A display is populated with representations of one or more options for operations associated with the vehicle. Based at least in part on an analysis of one or more signals generated by the occupant, a particular option is selected for implementation. An indication that the particular option has been selected is provided, and an operation corresponding to the particular option is initiated.

First claim

What is claimed is:

1. A method, comprising:
performing, using one or more computing devices:
detecting a gaze or a gesture from an occupant of the vehicle;
determining relevance scores for individual ones of a plurality of options for operations associated with the vehicle, wherein at least one option of the plurality of options comprises a destination that is determined based at least on:
the gaze or the gesture being directed from the occupant toward the destination; and
analysis of one or more signals obtained from the environment external to the vehicle;

populating a display with representations of the plurality of options for operations associated with the vehicle based at least on the determined relevance scores;
selecting, based at least on receiving input from the occupant of the vehicle, a particular option of the plurality of options, the input indicative of a voice command and another gesture; and
causing initiation of an operation corresponding to the particular option.

2. The method as recited in claim 1 , further comprising:
detecting that a triggering condition has been met for initiation of an interaction session with respect to the occupant of the vehicle, wherein said detecting that the triggering condition has been met comprises determining one or more of: (a) that at least a portion of a limb of the occupant has entered an interaction zone or (b) that at least a portion of a limb of the occupant has remained within an interaction zone for a threshold time interval.

3. The method as recited in claim 2 , wherein said populating is performed subsequent to detecting that a triggering condition has been met for initiation of the interaction session with respect to the occupant of the vehicle, wherein said detecting that the triggering condition has been met comprises detecting one or more of: (a) a voice command, (b) the direction of the gaze, or (c) a body movement of the occupant.

4. The method as recited in claim 2 , wherein said populating is performed subsequent to detecting that a triggering condition has been met for initiation of the interaction session with respect to the occupant of the vehicle, the method further comprising performing, by the one or more computing devices:
identifying at least a first option of the plurality of options prior to detecting that the triggering condition has been met; and
providing, prior to detecting that the triggering condition has been met, at least a de-emphasized representation of the first option via the display; and
providing, after detecting that the triggering condition has been met, an enhanced representation of the first option on the display.

5. The method as recited in claim 2 , further comprising performing, by the one or more computing devices:
in response to detecting that a triggering condition has been met for termination of the gesture-based interaction session, discontinuing receiving the input from the occupant.

6. The method as recited in claim 1 , wherein the display comprises one or more of: (a) a heads-up display incorporated within or attached to the vehicle, (b) a three-dimensional display, (c) a display of a wearable device being worn by the occupant or (d) a display of a portable computing device.

7. The method as recited in claim 1 , wherein said analysis of the other gesture comprises identifying a direction of a displacement of a hand of the occupant relative to an initial position of the hand.

8. The method as recited in claim 1 , further comprising providing an indication that the particular option has been selected comprising one or more of: (a) highlighting a representation of the particular option on the display, (b) providing an audio message, or (c) displaying a text string.

9. The method as recited in claim 1 , wherein the operation corresponding to the selected option comprises one of: (a) passing another vehicle, (b) accelerating the vehicle, (c) decelerating the vehicle, (d) parking the vehicle, (e) changing a direction in which the vehicle is moving, or (f) generating a signal detectable outside the vehicle.

10. A system, comprising:
one or more sensors; and
one or more interaction management devices;
wherein the one or more interaction management devices are configured to:
detect a gaze or a gesture from an occupant of a vehicle;
determine relevance scores for individual ones of a plurality of options for operations associated with the vehicle, wherein at least one option of the plurality of options comprises a destination that is determined based at least on:
the gaze or the gesture being directed from the occupant toward the destination; and
analysis of one or more signals obtained from the environment external to the vehicle;

populate a display with representations of a plurality of options for operations associated with the vehicle based at least on the determined relevance scores;
select, based on receiving input from the occupant of the vehicle, a particular option of the plurality of options, the input indicative of a voice command and another gesture; and
cause initiation of an operation corresponding to the particular option.

11. The system as recited in claim 10 , wherein the one or more interaction management devices are configured to detect, based at least in part on data collected at the one or more sensors, that a triggering condition has been met for initiation of an interaction session with respect to the occupant of the vehicle, wherein to detect that the triggering condition has been met, the one or more interaction management devices are configured to:
determine one or more of: (a) that at least a portion of a limb of the occupant has entered an interaction zone or (b) that at least a portion of a limb of the occupant has remained within an interaction zone for a threshold time interval.

12. The system as recited in claim 10 , wherein the display comprises one or more of: (a) a heads-up display incorporated within or attached to the vehicle, (b) a three-dimensional display, (c) a display of a wearable device being worn by the occupant or (d) a display of a portable computing device.

13. The system as recited in claim 10 , wherein the operation corresponding to the particular option comprises one of: (a) passing another vehicle, (b) accelerating the vehicle, (c) decelerating the vehicle, (d) parking the vehicle, (e) changing a direction in which the vehicle is moving, or (f) generating a signal detectable outside the vehicle.

14. The system as recited in claim 10 , wherein the one or more interaction management devices are configured to:
provide an indication that the particular option has been selected; and
verify, prior to causing the operation corresponding to the particular option to be initiated, that the operation has been approved by the occupant, wherein said verifying is based at least in part on an analysis of one or more signals detected after the indication that the particular option has been selected has been provided.

15. A non-transitory computer-accessible storage medium storing program instructions that when executed on one or more processors:
detect a gaze or a gesture from an occupant of a vehicle;
determine relevance scores for individual ones of a plurality of options associated with operations of the vehicle, wherein the plurality of options include one or more destinations determined based at least on the gaze or the gesture being directed from the occupant toward the one or more destinations;
populate a display with representations of the plurality of options for operations associated with the vehicle based at least on the determined relev

Assignees

• Apple Inc

Inventors

• Herz Scott M
• Bark Karlin
• Le Nguyen-Cat

Classifications

• B60K35/22

  Display screens · CPC title

• B60K35/26

  using acoustic output · CPC title

• B60K35/235

  with means for detecting the driver's gaze direction or eye points · CPC title

• B60K35/10

  Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor · CPC title

• G06F3/0482 Primary

  Interaction with lists of selectable items, e.g. menus · CPC title