Environment interactive system providing augmented reality for in-vehicle infotainment and entertainment

Patent metadata

Field	Value
Publication number	US-11617941-B2
Application number	US-202017008758-A
Country	US
Kind code	B2
Filing date	Sep 1, 2020
Priority date	Sep 1, 2020
Publication date	Apr 4, 2023
Grant date	Apr 4, 2023

Abstract

An environment interactive system includes: a first camera capturing first images of an area external to the vehicle; a second camera capturing second images of an area interior to the vehicle; a telematics control module determining a geographical location of the vehicle; a viewing device displaying items adjacent to, on or in front of detected objects and in a FOV of an occupant. The items correspond to the detected objects; and an infotainment module. The infotainment module: detects the objects based on the first images and the location of the vehicle; based on the second images, tracks at least one of a location of an eye of the occupant or a location of a body part of the occupant other than the eye; and based on the location of the eye and the location of the body part, displays the items via the viewing device in the FOV.

First claim

What is claimed is:

1. An environment interactive system for a vehicle, the environment interactive system comprising:
a first camera configured to capture first images of an area external to the vehicle;
a second camera configured to capture second images of an area interior to the vehicle;
a telematics control module configured to determine a geographical location of the vehicle;
a viewing device configured to display items (i) at least one of adjacent to, on or in front of detected objects, which are external to the vehicle, and (ii) in a field of view of an occupant, wherein the items include information corresponding to the detected objects; and
an infotainment module configured to
detect the objects based on the first images and the geographical location of the vehicle,
based on the second images, track at least one of a location of an eye of the occupant or a location of a body part of the occupant other than the eye,
based on the location of the eye and the location of the body part, display the items via the viewing device in the field of view of the occupant,
receive information and preferences of the occupant,
display an item for the occupant to select on the viewing device corresponding to starting a game, and
based on the location of the eye and the location of the body part, start the game.

2. The environment interactive system of claim 1 , further comprising a third camera configured to capture third images of an interior of the vehicle,
wherein the infotainment module is configured to, based on the second images and the third images, track the at least one of the location of the eye of the occupant or the body part of the occupant.

3. The environment interactive system of claim 1 , wherein:
the infotainment module is configured to determine a gaze point of the eye of the occupant; and
based on the location of the gaze point and the location of the body part, display the items via the viewing device in the field of view of the occupant.

4. The environment interactive system of claim 1 , the infotainment module is configured to, based on at least one of the location of the eye or the location of the body part, display the items in front of the objects and in the field of view of the occupant.

5. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
receive information and preferences of the occupant; and
based on the information and preferences, display information about certain ones of the objects in the field of view of the occupant.

6. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
receive information and preferences of the occupant; and
based on the information and preferences, display at least one of website pages or applications related to certain ones of the objects in the field of view of the occupant.

7. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
receive information and preferences of the occupant; and
based on the information and preferences, display at least one of targeted advertisements or coupons related to certain ones of the objects in the field of view of the occupant.

8. The environment interactive system of claim 1 , wherein the game includes displaying questions in the field of view of the occupant for the occupant to read and answer.

9. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
determine if the vehicle has stopped at a recommended location of a business; and
display in the field of view of the occupant a survey question for the occupant to answer pertaining to the business.

10. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
detect when the occupant selects one of the objects by gesturing at the selected one of the objects; and
display information pertaining to the selected one of the objects.

11. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
display information on a head up display pertaining to one of the objects; and
based on the at least one of the location of the eye or the location of the body part, detect when the occupant selects the one of the objects.

12. The environment interactive system of claim 1 , wherein the infotainment module is configured to:
interpret a gesture made by the occupant with regards to at least one of one of the objects or one of the items; and
perform an action in response to the gesture including enabling selection of at least one of the items or the objects.

13. The environment interactive system of claim 1 , wherein the infotainment module is configured to interpret a gesture or voice response provided by the occupant to confirm whether eye tracking of a gaze point of the occupant is accurate.

14. The environment interactive system of claim 1 , wherein the game is an environment interactive game played in an augmented reality.

15. The environment interactive system of claim 1 , wherein the game includes displaying trivia questions, via the viewing device, related to at least one of an environment external to the vehicle and an object in the environment external to the vehicle.

16. The environment interactive system of claim 15 , wherein the game includes feedback regarding answers to the trivia questions via at least one of an audible device and a haptic device.

17. An environment interactive method for an occupant of a vehicle, the environment interactive method comprising:
capturing first images of an area external to the vehicle;
capturing second images of an area interior to the vehicle;
determining a geographical location of the vehicle;
displaying items (i) between detected objects and the occupant, and (ii) in a field of view of the occupant, wherein the items correspond to the detected objects;
detecting the objects based on the first images and the geographical location of the vehicle;
based on the second images, tracking a location of an eye of the occupant and a location of a body part of the occupant other than the eye;
based on the location of the eye and the location of the body part, displaying the items via a viewing device in the field of view of the occupant,
wherein the viewing device is configured, based on user preferences, to display targeted information for one or more businesses at one or more locations;
detecting a first one or more gestures by the occupant for selection of one of the one or more locations;
based on a second one or more gestures from the occupant, display via the viewing device or another display device navigation instructions to the selected one of the one or more locations, the navigation instructions being provided in an augmented reality such that the navigation instructions are provided in the field of view of the occupant and include directions for driving the vehicle to the selected one of the one or more locations, the field of view including an environment external to the vehicle; and
at least one
i) display an item for the occupant to select on the viewing device corresponding to starting a game and, based on the location of the eye and the location of the body part, start the game, and
ii) determine a business where the vehicle is stopped and take a survey of the occupant via the viewing device for the business where the vehicle is stopped.

18. The method of claim 17 , further comprising:
tracking an iris of the eye of the occupant; and
based on the location of the

Assignees

• Gm Global Tech Operations Llc

Inventors

• Zeng Wei
• Dernotte Jeremie
• Liu Ke
• Ellison Nicole
• Mehdi Syed B
• Silang Kevin J
• Hector Jr Louis G

Classifications

• B60K35/10

  Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor · CPC title

• B60K35/235

  with means for detecting the driver's gaze direction or eye points · CPC title

• B60K35/28

  characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title