Systems and methods for artificial intelligence-based virtual and augmented reality

Patent metadata

Field	Value
Publication number	US-11315325-B2
Application number	US-201916596610-A
Country	US
Kind code	B2
Filing date	Oct 8, 2019
Priority date	Oct 9, 2018
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

Examples of the disclosure describe systems and methods for generating and displaying a virtual companion. In an example method, a first input from an environment of a user is received at a first time via a first sensor on a head-wearable device. An occurrence of an event in the environment is determined based on the first input. A second input from the user is received via a second sensor on the head-wearable device, and an emotional reaction of the user is identified based on the second input. An association is determined between the emotional reaction and the event. A view of the environment is presented at a second time later than the first time via a see-through display of the head-wearable device. A stimulus is presented at the second time via a virtual companion displayed via the see-through display, wherein the stimulus is determined based on the determined association between the emotional reaction and the event.

First claim

The invention claimed is:

1. A method comprising:
receiving, at a first time, via a first sensor on a head-wearable device, a first input from an environment of a user;
determining, based on the first input, an occurrence of an event in the environment;
receiving, via a second sensor on the head-wearable device, a second input from the user;
identifying, based on the second input, an emotional reaction of the user, wherein the emotional reaction is identified from a group of two or more different types of emotional reactions;
determining an association between the emotional reaction and the event;
receiving, at a second time later than the first time, an indication of an attempt by the user to recall information associated with the event, the indication received via the second sensor;
constructing a query based on the indication;
presenting, at the second time, via a see-through display of the head-wearable device, a view of the environment; and
presenting, at the second time, via a virtual companion displayed via the see-through display, a response to the query, wherein the response is determined based on the determined association between the emotional reaction and the event.

2. The method of claim 1 , wherein the first input comprises an image of a physical object.

3. The method of claim 1 , wherein the first input comprises an audio signal.

4. The method of claim 1 , wherein the second input comprises speech from the user, and wherein identifying the emotional reaction includes determining a content for at least a portion of the speech.

5. The method of claim 1 , wherein the second input comprises an eye movement of the user, and wherein identifying the emotional reaction includes determining a gaze direction for the user.

6. The method of claim 1 , wherein the second input comprises a field of view of the user, and wherein identifying the emotional reaction includes identifying at least one object within the field of view.

7. The method of claim 1 , the method further comprising determining an intensity of the emotional reaction, wherein the response to the query is determined further based on the intensity.

8. The method of claim 1 , wherein the association between the emotional reaction and the event is a temporal association.

9. The method of claim 1 , wherein the association between the emotional reaction and the event is a spatial association.

10. The method of claim 1 , wherein the event is a first event, the method further comprising storing the association between the emotional reaction and the first event in a memory graph, wherein the memory graph comprises an association between the first event and a second event.

11. A system comprising:
a first sensor on a head-wearable device;
a second sensor on the head-wearable device;
a see-through display of the head-wearable device;
one or more processors configured to execute a method comprising:
receiving, at a first time, via the first sensor on a head-wearable device, a first input from an environment of a user;
determining, based on the first input, an occurrence of an event in the environment;
receiving, via the second sensor on the head-wearable device, a second input from the user;
identifying, based on the second input, an emotional reaction of the user, wherein the emotional reaction is identified from a group of two or more different types of emotional reactions;
determining an association between the emotional reaction and the event;
receiving, at a second time later than the first time, an indication of an attempt by the user to recall information associated with the event, the indication received via the second sensor;
constructing a query based on the indication;
presenting, at the second time, via the see-through display of the head-wearable device, a view of the environment; and
presenting, at the second time, via a virtual companion displayed via the see-through display, a response to the query, wherein the response is determined based on the determined association between the emotional reaction and the event.

12. The system of claim 11 , wherein the first input comprises an image of a physical object.

13. The system of claim 11 , wherein the first input comprises an audio signal.

14. The system of claim 11 , wherein the second input comprises speech from the user, and wherein identifying the emotional reaction includes determining a content for at least a portion of the speech.

15. The system of claim 11 , wherein the second input comprises an eye movement of the user, and wherein identifying the emotional reaction includes determining a gaze direction for the user.

16. The system of claim 11 , wherein the second input comprises a field of view of the user, and wherein identifying the emotional reaction includes identifying at least one object within the field of view.

17. The system of claim 11 , the method further comprising determining an intensity of the emotional reaction, wherein the response to the query is determined further based on the intensity.

18. The system of claim 11 , wherein the association between the emotional reaction and the event is a temporal association.

19. The system of claim 11 , wherein the association between the emotional reaction and the event is a spatial association.

20. The system of claim 11 , wherein the event is a first event, the method further comprising storing the association between the emotional reaction and the first event in a memory graph, wherein the memory graph comprises an association between the first event and a second event.

21. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to execute a method comprising:
receiving, at a first time, via a first sensor on a head-wearable device, a first input from an environment of a user;
determining, based on the first input, an occurrence of an event in the environment;
receiving, via a second sensor on the head-wearable device, a second input from the user;
identifying, based on the second input, an emotional reaction of the user, wherein the emotional reaction is identified from a group of two or more different types of emotional reactions;
determining an association between the emotional reaction and the event;
receiving, at a second time later than the first time, an indication of an attempt by the user to recall information associated with the event, the indication received via the second sensor;
constructing a query based on the indication;
presenting, at the second time, via a see-through display of the head-wearable device, a view of the environment; and
presenting, at the second time, via a virtual companion displayed via the see-through display, a response to the query, wherein the response is determined based on the determined association between the emotional reaction and the event.

22. The non-transitory computer-readable medium of claim 21 , wherein the first input comprises an image of a physical object.

23. The non-transitory computer-readable medium of claim 21 , wherein the first input comprises an audio signal.

24. The non-transitory computer-readable medium of claim 21 , wherein the second input comprises speech from the user, and wherein identifying the emotional reaction includes determining a content for at least a portion of the speech.

25. The non-transitory computer-readable medium of claim 21

Assignees

• Magic Leap Inc

Inventors

• Rabinovich Andrew
• Monos John

Classifications

• G06F2203/011

  Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F2203/012

  Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment · CPC title

• H04S2420/01

  Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title