Image generation based on brain activity monitoring

Patent metadata

Field	Value
Publication number	US-10706631-B2
Application number	US-201816176578-A
Country	US
Kind code	B2
Filing date	Oct 31, 2018
Priority date	Oct 31, 2018
Publication date	Jul 7, 2020
Grant date	Jul 7, 2020

Abstract

Systems, methods, and devices for generating an image frame for display to a user. Brain activity sensor data correlated with movement of a user is received. A predicted field of view of the user is determined based on the brain activity sensor data. An image frame is generated based on the predicted field of view. The image frame is transmitted to a display for display to a user. Some implementations provide for receiving and displaying a foveated image frame based on a predicted field of view of a user. Brain activity information of a user is captured. The brain activity information is communicated to a transceiver. The brain activity information is transmitted to a rendering device using the transceiver to generate a foveated image frame based on a predicted field of view of the user. The foveated image frame is received from the rendering device, decoded, and displayed to the user.

First claim

What is claimed is:

1. A method for generating an image frame for display to a user, the method comprising:
receiving brain activity sensor data correlated with movement of a user;
determining a predicted gaze direction of the user based on the brain activity sensor data;
generating an image frame based on the predicted gaze direction; and
transmitting the image frame to a display.

2. The method of claim 1 , wherein generating the predicted gaze direction comprises:
inputting the brain activity sensor data to an artificial neural network (ANN); and
determining the predicted gaze direction of the user using the ANN based on the input brain activity sensor data.

3. The method of claim 1 , wherein generating the image frame based on the predicted gaze direction comprises:
determining a predicted field of view of the user based on the gaze direction; and
rendering graphics data to generate the image frame with a first fidelity in an area of the image frame corresponding to a center of the predicted field of view, and with a second fidelity lower than the first fidelity in an area of the image frame not corresponding to the center of the predicted field of view.

4. The method of claim 1 , wherein generating the image frame based on the predicted gaze direction comprises:
determining a predicted field of view of the user based on the gaze direction; and
encoding the image frame with a first bit rate in an area of the image frame corresponding to a center of the predicted field of view, and with a second bit rate lower than the first bit rate in an area of the image frame not corresponding to the center of the predicted field of view.

5. The method of claim 1 , further comprising receiving eye tracking data correlated with a current gaze direction of the user; and generating the predicted gaze direction based on both the brain activity sensor data and the eye tracking data.

6. The method of claim 1 , further comprising receiving information correlated with a position, an orientation, or both the position and orientation of the user; and generating the predicted gaze direction based on both the brain activity sensor data and the information.

7. A device for generating an image frame for display to a user, the device comprising:
receiver circuitry configured to receive brain activity sensor data correlated with movement of a user;
processing circuitry configured to determine a predicted gaze direction of the user based on the brain activity sensor data;
processing circuitry configured to generate an image frame based on the predicted gaze direction; and
transmitter circuitry configured to transmit the image frame to a display.

8. The device of claim 7 , further comprising:
an artificial neural network (ANN) configured to input the brain activity sensor data and to determine the predicted gaze direction of the user based on the input brain activity sensor data.

9. The device of claim 7 , wherein the processing circuitry is further configured to determine a predicted field of view of the user based on the gaze direction; and
the device further comprises:
graphics processing circuitry configured to render graphics data to generate the image frame with a first fidelity in an area of the image frame corresponding to a center of the predicted field of view, and with a second fidelity lower than the first fidelity in an area of the image frame not corresponding to the center of the predicted field of view.

10. The device of claim 7 , wherein the processing circuitry is further configured to determine a predicted field of view of the user based on the gaze direction; and
the device further comprises:
encoder circuitry configured to encode the image frame with a first bit rate in an area of the image frame corresponding to a center of the predicted field of view, and with a second bit rate lower than the first bit rate in an area of the image frame not corresponding to the center of the predicted field of view.

11. The device of claim 7 , further comprising:
processing circuitry configured to receive eye tracking data correlated with a current gaze direction of the user; and to generate the predicted gaze direction based on both the brain activity sensor data and the eye tracking data.

12. The device of claim 7 , further comprising receiving information correlated with a position, an orientation, or both the position and orientation of the user; and generating the predicted gaze direction based on both the brain activity sensor data and the information.

13. A method for receiving and displaying a foveated image frame based on a predicted gaze direction of a user, the method comprising:
capturing brain activity information of a user;
communicating the brain activity information to a transceiver;
transmitting the brain activity information using the transceiver, to a rendering device to predict a gaze direction based on the brain activity information and to generate a foveated image frame based on the predicted gaze direction of the user;
receiving the foveated image frame from the rendering device;
decoding the foveated image frame; and
displaying the foveated image frame to the user.

14. The method of claim 13 , wherein the foveated image frame comprises a first fidelity in an area of the image frame corresponding to a center of a predicted field of view generated based on the predicted gaze direction, and a second fidelity lower than the first fidelity in an area of the image frame not corresponding to the center of the predicted field of view.

15. The method of claim 13 , wherein the foveated image frame is encoded with a first bit rate in an area of the foveated image frame corresponding to a center of a predicted field of view generated based on the predicted gaze direction, and with a second bit rate lower than the first bit rate in an area of the foveated image frame not corresponding to the center of the predicted field of view.

16. The method of claim 13 , further comprising capturing eye tracking information correlated with a current gaze direction of the user; and transmitting the eye tracking information to the rendering device to generate a predicted gaze direction based on the brain activity information and the eye tracking information.

17. The method of claim 13 , further comprising capturing information correlated with a position, an orientation, or both the position and orientation of the user; and transmitting the information to the rendering device to generate the predicted gaze direction based on the brain activity information and the information.

18. The method of claim 13 , further comprising capturing the brain activity information of the user using an electroencephalography (EEG) sensor.

19. The method of claim 16 , further comprising capturing the eye tracking data using a camera mounted in a head-mounted display (HMD).

20. The method of claim 17 , further comprising capturing the information using at least one accelerometer.

Assignees

• Advanced Micro Devices Inc

Inventors

• Schmit Michael L
• Naegle Nathaniel David

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G09G2340/0407

  Resolution change, inclusive of the use of different resolutions for different screen areas · CPC title

• G06F3/015

  Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title