Artificial reality system with varifocal display of artificial reality content

Patent metadata

Field	Value
Publication number	US-11217024-B2
Application number	US-201916727167-A
Country	US
Kind code	B2
Filing date	Dec 26, 2019
Priority date	Dec 26, 2019
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

The disclosure describes artificial reality systems and techniques for providing artificial reality content to a user. For example, an artificial reality system includes a head-mounted display (HMD) configured to output artificial reality content, the HMD including a set of second image capture devices configured to capture image data indicative of a focal point of a gaze of the user and a varifocal display having a focal length that is modifiable based on the focal point of the user. Additionally, the system includes a depth engine configured to generate, based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the physical environment of the user and generate artificial reality content as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD based on the focal point of the user.

First claim

What is claimed is:

1. An artificial reality system comprising:
a first image capture device configured to capture real-world image data representative of a physical environment of a user;
a head-mounted display (HMD) configured to output artificial reality content, the HMD comprising:
a set of second image capture devices configured to capture image data indicative of a focal point of a gaze of the user;
a varifocal display having a focal length that is modifiable based on the focal point of the user; and
a motor configured to control a position of the varifocal display to modify the focal length; and

a depth engine configured to:
 determine the focal length for the varifocal display in near real-time to match the focal point of the user;
 control the motor in order to move the varifocal display along a longitudinal axis of the HMD relative to a lens to achieve the determined focal length;
 generate, based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the physical environment of the user; and
 generate artificial reality content as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD based on the focal point of the user.

2. The system of claim 1 , wherein the first image capture device comprises a pass-through camera configured to capture the real-world image data in color and at a high resolution.

3. The system of claim 2 , wherein the HMD comprises one or more processors configured to execute the depth engine, wherein the one or more processors are located proximate to the at least one pass-through camera.

4. The system of claim 1 , wherein the HMD further comprises a depth sensor configured to generate the depth data, and wherein to generate the 3D scene of the physical environment, the depth engine is configured to:
generate, using the depth data, a 3D mesh of a real-world scene; and
overlay at least a portion of the real-world image data onto the 3D mesh of the real-world scene.

5. The system of claim 4 , wherein the real-world image data represents a stream of real-world image data, wherein the depth data represents a stream of depth data, and wherein to generate the artificial reality content, the depth engine is further configured to:
generate, using the stream of real-world image data and the stream of depth data, the artificial reality content in near real-time based on a position and an orientation of the HMD with respect to the 3D mesh of the real-world scene.

6. The system of claim 1 , wherein the set of second image capture devices comprises:
a first eye-tracking camera configured to capture a first set of image data including a position of a first pupil of the user in relation to a first set of reference points; and
a second eye-tracking camera configured to capture a second set of image data including a position of a second pupil of the user in relation to a second set of reference points, and wherein the depth engine is configured to:
determine the focal point of the user based on the position of the first pupil in relation to the first set of reference points and the position of the second pupil in relation to the second set of reference points.

7. The system of claim 6 , further comprising a gaze tracker configured to:
track the position of the first pupil in relation to the first set of reference points over a period of time;
track the position of the second pupil in relation to the second set of reference points over the period of time;
determine, based on a movement of the position of the first pupil in relation to the first set of reference points over the period of time, a projected future movement of the position of the first pupil in relation to the first set of reference points; and
determine, based on a movement of the position of the second pupil in relation to the second set of reference points over the period of time, a projected future movement of the position of the second pupil in relation to the second set of reference points,
wherein the depth engine is further configured to determine, based on the projected future movement of the position of the first pupil and the projected future movement of the position of the second pupil, a projected future focal point of the user.

8. The system of claim 1 , wherein the image data is further indicative of a depth of field of the user, and wherein to generate the artificial reality content, the depth engine is configured to:
blur portions of the artificial reality content that are outside of the depth of field of the user.

9. A method comprising:
capturing, by a first image capture device, real-world image data representative of a physical environment of a user;
capturing, by a set of second image capture devices of a head-mounted display (HMD) configured to output artificial reality content, image data indicative of a focal point of a gaze of the user;
modifying, by a depth engine of the HMD, a focal length of a varifocal display of the HMD based on the focal point of the user, wherein modifying the focal length of the varifocal display comprises:
determining, by the depth engine, the focal length for the varifocal display in near real-time to match the focal point of the user; and
controlling, by the depth engine, a motor in order to move the varifocal display along a longitudinal axis of the HMD relative to a lens to achieve the determined focal length;

generating, by the depth engine and based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the physical environment of the user; and
generating, by the depth engine, artificial reality content as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD based on the focal point of the user.

10. The method of claim 9 , wherein the first image capture device comprises a pass-through camera, and wherein the method further comprises capturing, using the pass-through camera, the real-world image data in color and at a high resolution.

11. The method of claim 10 , further comprising executing, using one or more processors of the HMD, the depth engine, wherein the one or more processors are located proximate to the at least one pass-through camera.

12. The method of claim 9 , further comprising:
generating, using a depth sensor, the depth data, and wherein generating the 3D scene of the physical environment comprises:
generating, using the depth data, a 3D mesh of a real-world scene; and
overlaying at least a portion of the real-world image data onto the 3D mesh of the real-world scene.

13. The method of claim 12 , wherein the real-world image data represents a stream of real-world image data, wherein the depth data represents a stream of depth data, and wherein generating the artificial reality content comprises:
generating, using the stream of real-world image data and the stream of depth data, the artificial reality content in near real-time based on a position and an orientation of the HMD with respect to the 3D mesh of the real-world scene.

14. The method of claim 9 , further comprising:
capturing, using a first eye-tracking camera of the set of second image capture devices, a first set of image data including a position of a first pupil of the user in relation to a first set of reference points;
capturing, using a second eye-tracking camera of the set of second image capture devices, a set of second image data including a position of a second pupil of the user in relation to a second set of reference points; and
determining

Assignees

• Facebook Tech Llc

Inventors

• Sztuk Sebastian
• Khan Saara

Classifications

• G06V40/19

  Sensors therefor · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title