Augmented reality vehicle user interface

Patent metadata

Field	Value
Publication number	US-10410426-B2
Application number	US-201715847433-A
Country	US
Kind code	B2
Filing date	Dec 19, 2017
Priority date	Dec 19, 2017
Publication date	Sep 10, 2019
Grant date	Sep 10, 2019

Abstract

A system and method of operating a vehicle using virtual vehicle controls, wherein the method includes: capturing image or video data from an area within an interior of the vehicle; sending a virtual vehicle control graphics request to a vehicle; receiving virtual vehicle control graphics response from the vehicle, wherein the virtual vehicle control graphics response includes virtual vehicle control graphics; and presenting the captured image or video data and the virtual vehicle control graphics on a display of the AR/VR device such that the virtual vehicle control graphics are presented over the captured image or video data, wherein the virtual vehicle control graphics includes one or more vehicle-user interface components.

First claim

The invention claimed is:

1. A method of operating a vehicle using virtual vehicle controls, wherein the method comprises:
establishing a SRWC connection between an augmented reality/virtual reality (AR/VR) device and a wireless communications device included in vehicle electronics of a vehicle using short-range wireless communications (SRWC);
capturing image or video data from an area within an interior of the vehicle;
sending a virtual vehicle control graphics request to the vehicle via the established SRWC connection;
receiving a virtual vehicle control graphics response from the vehicle via the established SRWC connection, wherein the virtual vehicle control graphics response includes virtual vehicle control graphics;
presenting the captured image or video data and the virtual vehicle control graphics on a display of the AR/VR device such that the virtual vehicle control graphics are presented over the captured image or video data, wherein the virtual vehicle control graphics includes one or more vehicle-user interface components;
receiving vehicle control input from a user at the AR/VR device, wherein the vehicle control input indicates a selection or modification of the one or more vehicle-user interface components, wherein the vehicle control input is initially received at a camera of the AR/VR device, and wherein the vehicle control input is communicated by the user to the AR/VR device via one or more gestures created by the user in a field of view of the camera; and
sending the vehicle control input to the vehicle via the established SRWC connection, wherein the vehicle is configured to carry out one or more actions based on the vehicle control input.

2. The method of claim 1 , wherein the captured image or video data is captured by the camera located on the AR/VR device, and wherein the camera of the AR/VR device faces an area in front of a user when the display of the AR/VR device is positioned to face the user.

3. The method of claim 2 , wherein the AR/VR device is a pair of AR/VR goggles configured for visually presenting augmented reality or virtual reality media content to a user and that are wearable by the user and, when the AR/VR device is worn properly by the user, the display of the AR/VR device faces the user and the camera of the AR/VR device captures an area in front of the user.

4. The method of claim 1 , further comprising the step of receiving a virtual vehicle control graphics advertisement that indicates virtual vehicle control graphics are available for use by the AR/VR device.

5. The method of claim 1 , wherein the SRWC connection is established using BLUETOOTH, WI-FI, or other IEEE 802.11 technology.

6. The method of claim 1 , wherein the virtual vehicle control graphics response includes virtual vehicle control graphics location information that indicates a vehicle component that the virtual vehicle control graphics are to be displayed on or that indicates a location on the display of the AR/VR device where the virtual vehicle control graphics are to be displayed.

7. The method of claim 1 , further comprising the steps of:
receiving a second virtual vehicle control graphics response from the vehicle after sending the vehicle control input to the vehicle, and wherein the second virtual vehicle control graphics includes updated virtual vehicle control graphics; and
presenting the updated virtual vehicle control graphics on the display of the AR/VR device.

8. The method of claim 1 , wherein the virtual vehicle control graphics are updated based on the vehicle control input.

9. The method of claim 1 , wherein the vehicle control input is received at the vehicle from the AR/VR device.

10. The method of claim 9 , wherein the vehicle control input is initially received at the AR/VR device using a controller of the AR/VR device.

11. The method of claim 9 , further comprising the step of authorizing the vehicle control input based on authorization information contained within the vehicle control input and/or information contained in the virtual vehicle control graphics request.

12. A method of controlling a vehicle using virtual vehicle controls, wherein the method comprises:
establishing a short-range wireless communications (SRWC) connection to an augmented reality/virtual reality (AR/VR) device, wherein the AR/VR device includes a display and a camera, wherein the AR/VR device is configured to capture image or video data, and wherein the AR/VR device includes short-range wireless communication (SRWC) circuitry that is used to send and receive SRWC signals, wherein the AR/VR device is separate from vehicle electronics of the vehicle;
receiving a virtual vehicle control graphics request from the AR/VR device;
determining whether the AR/VR device is authorized and/or authenticated for receiving and/or using virtual vehicle control graphics;
sending virtual vehicle control graphics response to the AR/VR device via the established SRWC connection, wherein the AR/VR device is further configured to present the virtual vehicle control graphics on the display of the AR/VR device such that the virtual vehicle control graphics are presented over the captured image or video data, wherein the at least one of the virtual vehicle control graphics is presented on a display of the AR/VR device over the captured image or video data, wherein the at least one virtual vehicle control graphic is located within the display of the AR/VR device at a location that corresponds to a location within a cabin of the vehicle in which a capacitive touch sensor array of the vehicle is provided, and wherein the capacitive touch sensor array is included as a part of the vehicle electronics of the vehicle;
receiving vehicle control input from a user of the AR/VR device, wherein the receiving step further includes receiving sensor location input from the capacitive touch sensor array, and wherein the sensor location input indicates a location within the capacitive touch sensor array that is touched by the user;
determining a vehicle action to be carried out based on the sensor location input and the virtual vehicle control graphics; and
carrying out the vehicle action at the vehicle.

13. The method of claim 12 , wherein the virtual vehicle control graphics response includes virtual vehicle control graphics location information that indicates a vehicle component that the virtual vehicle control graphics are to be displayed on or that indicates a location within the display of the AR/VR device where the virtual vehicle control graphics are to be displayed.

14. The method of claim 12 , wherein the vehicle control input indicates a selection or modification of the one or more vehicle-user interface components, and wherein the selection or modification is determined based on the sensor location input.

15. The method of claim 12 , further comprising the step of modifying operation of one or more vehicle system modules (VSMs) in response to receiving the vehicle control input and wherein the modifying step is based on the vehicle control input.

16. The method of claim 15 , further comprising the step of, after modifying operation of one or more vehicle system modules (VSMs), sending a vehicle control input response that includes information used for rendering updated virtual vehicle control graphics.

17. The method of claim 16 , wherein the information used for rendering the updated virtual vehicle control graphics includes an instance of the updated virtual vehicle control graphics, and wherein the updated virtual vehicle control graphics are generated at the vehicle based on the vehicle c

Assignees

• Gm Global Tech Operations Llc

Inventors

• Kamini Aditya S
• Williams Nathaniel H
• Elswick Richard

Classifications

• G09G2370/16

  Use of wireless transmission of display information · CPC title

• G09G2380/10

  Automotive applications · CPC title

• G06F3/14

  Digital output to display device {; Cooperation and interconnection of the display device with other functional units} · CPC title

• G09G2340/12

  Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels · CPC title

• G09G2354/00

  Aspects of interface with display user · CPC title