Method for VR sickness assessment considering neural mismatch model and the apparatus thereof

Patent metadata

Field	Value
Publication number	US-11699072-B2
Application number	US-201916700834-A
Country	US
Kind code	B2
Filing date	Dec 2, 2019
Priority date	Dec 13, 2018
Publication date	Jul 11, 2023
Grant date	Jul 11, 2023

Abstract

A virtual reality (VR) sickness assessment method according to an embodiment includes receiving virtual reality content, and quantitatively evaluating virtual reality sickness for the received virtual reality content using a neural network based on a pre-trained neural mismatch model. The evaluating of the virtual reality sickness may include predicting an expected visual signal for an input visual signal of the virtual reality content based on the neural mismatch model, extracting a neural mismatch feature between the predicted expected visual signal based on the neural mismatch model and an input visual signal for a corresponding frame of the virtual reality content corresponding to the expected visual signal, and evaluating a level of the virtual reality sickness based on the neural mismatch model and the extracted neural mismatch feature.

First claim

What is claimed is:

1. A virtual reality (VR) sickness assessment method, comprising:
receiving virtual reality content; and
quantitatively evaluating virtual reality sickness for the received virtual reality content using a neural network based on a pre-trained neural mismatch model that is trained by performing hostile learning between a spatio-temporal predictor and a spatio-temporal discriminator, wherein the evaluating of the virtual reality sickness includes:
predicting an expected visual signal for an input visual signal of the virtual reality content based on the neural mismatch model;
extracting a neural mismatch feature between the predicted expected visual signal based on the neural mismatch model and an input visual signal for a corresponding frame of the virtual reality content corresponding to the expected visual signal; and
evaluating a level of the virtual reality sickness based on the neural mismatch model and the extracted neural mismatch feature.

2. The virtual reality sickness assessment method of claim 1 , wherein the extracting of the neural mismatch feature includes extracting the neural mismatch feature based on difference information between the predicted expected visual signal and the input visual signal for the corresponding frame.

3. The virtual reality sickness assessment method of claim 1 , wherein the extracting of the neural mismatch feature includes obtaining a difference feature for difference information between the predicted expected visual signal and the input visual signal for the corresponding frame and a visual feature for the input visual signal for the corresponding frame and extracting the neural mismatch feature based on the difference feature and the visual feature.

4. The virtual reality sickness assessment method of claim 1 , wherein the evaluating of the virtual reality sickness further includes generating a difference map for the virtual reality content based on difference information between the expected visual signal predicted for each frame of the virtual reality content and the input visual signal for the corresponding frame of the virtual reality content corresponding to the expected visual signal.

5. The virtual reality sickness assessment method of claim 1 , wherein the neural network is trained based on a dataset including pieces of preset virtual reality content and a simulator sickness questionnaire (SSQ) score evaluated for each of the pieces of virtual reality content to form a learning model of the neural mismatch model.

6. A virtual reality sickness assessment (VRSA) method, comprising:
receiving virtual reality content; and
predicting an expected visual signal for an input visual signal of the VR content by using a first neural network based on a pre-trained first learning model that is trained by performing hostile learning between a spatio-temporal predictor and a spatio-temporal discriminator;
extracting a neural mismatch feature between the expected visual signal predicted and the input visual signal of a corresponding frame of the virtual reality content corresponding to the expected visual signal using a second neural network based on a pre-trained second learning model; and
evaluating a level of virtual reality sickness corresponding to the extracted neural mismatch feature using a third neural network based on a pre-third trained learning model,
wherein a neural mismatch model is formed by the first learning model, the second learning model, and the third learning model.

7. The virtual reality sickness assessment method of claim 6 , wherein the extracting of the neural mismatch feature includes extracting the neural mismatch feature based on difference information between the predicted expected visual signal and the input visual signal for the corresponding frame.

8. The virtual reality sickness assessment method of claim 6 , wherein the extracting of the neural mismatch feature includes obtaining a difference feature for difference information between the predicted expected visual signal and the input visual signal for the corresponding frame and a visual feature for the input visual signal for the corresponding frame and extracting the neural mismatch feature based on the difference feature and the visual feature.

9. The virtual reality sickness assessment method of claim 6 , further comprising:
generating a difference map for the virtual reality content based on difference information between the expected visual signal predicted for each frame of the virtual reality content and the input visual signal for the corresponding frame of the virtual reality content corresponding to the expected visual signal.

10. A virtual reality sickness assessment (VRSA) apparatus, comprising:
a receiver configured to receive virtual reality content; and
an evaluator configured to quantitatively evaluate virtual reality sickness for the received virtual reality content using a neural network based on a pre-trained neural mismatch model that is trained by performing hostile learning between a spatio-temporal predictor and a spatio-temporal discriminator, wherein the evaluator is configured to:
predict an expected visual signal for an input visual signal of the virtual reality content based on the neural mismatch model;
extract a neural mismatch feature between the predicted expected visual signal based on the neural mismatch model and an input visual signal for a corresponding frame of the virtual reality content corresponding to the expected visual signal; and
evaluate a level of the virtual reality sickness based on the neural mismatch model and the extracted neural mismatch feature.

11. The virtual reality sickness assessment apparatus of claim 10 , wherein the evaluator extracts the neural mismatch feature based on difference information between the predicted expected visual signal and the input visual signal for the corresponding frame.

12. The virtual reality sickness assessment apparatus of claim 10 , wherein the evaluator obtains a difference feature for difference information between the predicted expected visual signal and the input visual signal for the corresponding frame and a visual feature for the input visual signal for the corresponding frame and extracts the neural mismatch feature based on the difference feature and the visual feature.

13. The virtual reality sickness assessment apparatus of claim 10 , further comprising:
a generator configured to generate a difference map for the virtual reality content based on difference information between the expected visual signal predicted for each frame of the virtual reality content and the input visual signal for the corresponding frame of the virtual reality content corresponding to the expected visual signal.

14. The virtual reality sickness assessment apparatus of claim 10 , wherein the neural network is trained based on a dataset including pieces of preset virtual reality content and a simulator sickness questionnaire (SSQ) score evaluated for each of the pieces of virtual reality content to form a learning model of the neural mismatch model.

15. A virtual reality sickness assessment (VRSA) apparatus, comprising:
a receiver configured to receive virtual reality content; and
a predictor configured to predict an expected visual signal for an input visual signal of the VR content by using a first neural network based on a pre-trained first learning model that is trained by performing hostile learning against a discriminator;
an extractor configured to extract a neural mismatch feature between the expected visual signal predicted and the input visual signal of a corresponding frame of the

Assignees

• Korea Advanced Inst Sci & Tech

Inventors

• Ro Yongman
• Kim Hak Gu
• Lee Sangmin

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G09B3/06

  of the multiple-choice answer type, i.e. where a given question is provided with a series of answers and a choice has to be made · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title