Realistic rendering for virtual reality applications

Patent metadata

Field	Value
Publication number	US-10217286-B1
Application number	US-201514859984-A
Country	US
Kind code	B1
Filing date	Sep 21, 2015
Priority date	Sep 21, 2015
Publication date	Feb 26, 2019
Grant date	Feb 26, 2019

Abstract

Motion sickness resulting from use of a virtual reality headset can be mitigated by displaying a virtual nose in the field of view of the user. Nose data can be obtained by a user selecting various options, or determined dynamically using various image analysis algorithms. Image data captured of the user's face can be data analyzed to determine aspects such as the size, shape, color, texture, and reflectivity of the user's nose. A three-dimensional nose model is generated, which is treated as an object in the virtual world and can have lighting, shadows, and textures applied accordingly. The pupillary distance can be determined from the image data and used to determine the point of view from which to render each nose portion. Changes in lighting or expression can cause the appearance of the nose to change, as well as the level of detail of the rendering.

First claim

What is claimed is:

1. A virtual reality display system, comprising:
a display including a left eye region and a right eye region;
a processor; and
memory including instructions that, when executed by the processor, cause the virtual reality display system to:
obtain image data including a representation of a face of a user;
determine facial feature points based at least in part on the representation of the face in the image data;
determine, using the facial feature points, a shape of a nose of the user based at least in part on the image data;
determine, using the facial feature points, a pupillary distance of the user;
determine color data for a nose region of the image data, the nose region determined using the shape of the nose;
determine virtual reality content to be displayed by the display;
determine a left view of the nose to be displayed by the left eye region of the display, the left view determined using a left point of view of a left eye of the user, the shape of the nose, and the color data for the nose region;
determine a right view of the nose to be displayed by the right eye region of the display, the right view determined using a right point of view of a right eye of the user, the shape of the nose, and the color data for the nose region, a difference between the left eye point of view and the right eye point of view corresponding to the pupillary distance of the user; and
render the virtual reality content, left view of the nose, and right view of the nose for display.

2. The virtual reality display system of claim 1 , wherein the instructions when executed further cause the virtual reality display system to:
determine a relative position of a virtual light source in the virtual reality content, wherein the left view of the nose is rendered to reflect an effect of the virtual light source on a left portion of the user's nose, and wherein the right view of the nose is rendered to reflect an effect of the virtual light source on a right portion of the user's nose.

3. The virtual reality display system of claim 2 , further comprising:
a motion sensor configured to detect a change in orientation of the virtual reality display system, wherein the instructions when executed further cause the virtual reality display system to render a second left view of the nose reflecting a new effect of the virtual light source on the left portion of the nose for a new relative position of the virtual light source, and wherein a second right view of the nose is rendered that reflects a new effect of the virtual light source on the right portion of the nose for the new relative position of the virtual light source.

4. The virtual reality display system of claim 1 , wherein the instructions when executed further cause the virtual reality display system to:
generate a nose model representing the shape of the nose of the user; and
generate texture data using the color data for the nose region, the color data including a base color and a map of color deviation for different locations of the nose region,
wherein rendering the left view of the nose includes determining a left point of view of the left eye, determining a view of the nose model corresponding to the left point of view, and applying the texture data to the view of the nose model.

5. The virtual reality display system of claim 1 , wherein the instructions when executed further cause the virtual reality display system to:
determine, using a camera of the virtual reality display system, an expression of the user;
determine a second shape of the nose of the user corresponding to the expression;
render a second left view of the nose using the second shape; and
render a second right view of the nose using the second shape.

6. A computer-implemented method, comprising:
under control of one or more computer systems configured with executable instructions,
determining virtual reality content to be displayed to a user;
obtaining image data including a representation of the user;
determining a nose model for the user, the nose model based at least in part on the image data and including three-dimensional data representing a shape of a nose of the user;
determining appearance data for the nose of the user based at least in part on the image data;
determining a left view of the nose, the left view determined using a left point of view of a left eye of the user, the nose model, and the appearance data;
determining a right view of the nose, the right view determined using a right point of view of a right eye of the user, the nose model, and the appearance data;
causing a left view of the virtual reality content and the left view of the nose to be displayed via a left eye display of a display device; and
causing a right view of the virtual reality content and the right view of the nose to be displayed via a right eye display of the display device.

7. The computer-implemented method of claim 6 , further comprising:
determining a pupillary distance of the user, wherein a difference between the left view of the nose and the right view of the nose corresponds to the pupillary distance of the user.

8. The computer-implemented method of claim 6 , further comprising:
displaying a plurality of nose models;
receiving selection of the nose model from the plurality of nose models;
displaying a plurality of colors for the selected nose model; and
receiving a selection of a color of the plurality of colors, the color included in the appearance data.

9. The computer-implemented method of claim 6 , further comprising:
obtaining image data including a representation of a face of the user;
determining a set of facial feature points from the representation of the face in the image data; and
determining the nose model using the set of facial feature points, nose model representing a shape and size of the nose of the user.

10. The computer-implemented method of claim 9 , further comprising:
determining, from the set of facial feature points, a pupillary distance of the user.

11. The computer-implemented method of claim 9 , further comprising:
determining, using the set of facial feature points, a nose region of the image data;
determining a base color for the nose region, the base color representing an average intensity value for pixel locations within the nose region of the image data;
determining a variation from the base color for portion of the nose region; and
storing the base color and variation as the appearance data.

12. The computer-implemented method of claim 11 , further comprising:
analyzing a second set of image data obtained with an active source of illumination;
comparing second base color and second variation data for the second set of image data against the base color and variation to determine reflectivity data for the nose; and
storing the reflectivity data as part of the appearance data.

13. The computer-implemented method of claim 11 , further comprising:
displaying the left view of the nose with a level of detail corresponding to an intensity of a virtual light source.

14. The computer-implemented method of claim 6 , further comprising:
determining an expression of the user;
determining a second shape of the nose of the user corresponding to the expression;
rendering a second left view of the nose using the second shape; and
rendering a second right view of the nose using the second shape.

15. A computer-implemented method, comprising:
under control of one or more computer systems configured with executable instructions,
obtaining image data including a representation of a face of a user of

Assignees

• Amazon Tech Inc

Inventors

• Angel Justin-Josef
• Breitbard Eric Alan
• Swann Colin Neil
• Murdock Robert Steven

Classifications

• G02B27/017 Primary

  Head mounted · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title