Headware with computer and optical element for use therewith and systems utilizing same

Patent metadata

Field	Value
Publication number	US-11778149-B2
Application number	US-202117396976-A
Country	US
Kind code	B2
Filing date	Aug 9, 2021
Priority date	May 11, 2011
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

An apparatus for mounting on a head including a frame, A face-wearable near-ocular optics and a micro-display for displaying data in front of the eyes is provided. A computing device is coupled to the micro-display. At least one sensor is coupled to the computing device for receiving biometric human information.

First claim

What is claimed is:

1. An apparatus comprising:
a support structure for mounting on the head of a user;
a display matrix;
a plurality of optical lenses retained by the support structure, and disposed between the display matrix and an eye of the user, the plurality of optical lenses including a hemispherical element that covers at least a portion of the display matrix;
a microcomputer carried by the support structure;
a sensor device for receiving biometric data;
a camera mounted on the support structure; and
a transceiver coupled with the microcomputer, wherein the microcomputer executes instructions that comprise:
capturing a set of images by the camera mounted on the support structure;
generating an image stream based on the set of images;
extracting metadata from the image stream;
generating a request that includes the metadata to a networked computer;
receiving, from the networked computer, a presentation of the image stream that includes an object rendered at a position within the image stream based on at least the metadata and the biometric data;
causing the display matrix to display the presentation of the image stream that includes the object rendered at the position within the image stream upon the hemispherical element;
detecting biometric feedback; and
presenting the object within the presentation of the image stream based on the biometric feedback.

2. The apparatus of claim 1 , further comprising:
an inwardly-facing camera mounted on the support structure, the inwardly-facing camera producing eye tracking video of an eye that is processed by the microcomputer, wherein the microcomputer processes the eye tracking video to track pupil position, and wherein the generating the presentation of the image stream is based on the metadata and the eye tracking video.

3. The apparatus of claim 2 , wherein the microcomputer processes the eye tracking video to detect eye corners using oriented gradient features and averages the positions of the eye corners to obtain an x,y location for the eye.

4. The apparatus of claim 2 , wherein the microcomputer processes the eye tracking video to calculate a two-dimensional displacement between the center of the eye and the center of the pupil.

5. The apparatus of claim 2 , wherein the microcomputer processes the eye tracking video to calculate a two-dimensional displacement between the center of the eye and the center of the pupil, and wherein the microcomputer converts the two-dimensional displacement to a three-dimensional gaze vector for the eye.

6. The apparatus of claim 2 , wherein the pupil position is used to alter the position of the object rendered within the image stream.

7. The apparatus of claim 1 , wherein the microcomputer performs a calibration operation by projecting a sequence of objects on a display mounted on the support structure and collecting viewed object acknowledgements from a user.

8. A method comprising:
capturing a set of images by a camera mounted on a support structure of a head-mounted apparatus, the head-mounted apparatus comprising a display matrix, and a plurality of optical lenses retained by the support structure and disposed between the display matrix and an eye of a user;
generating an image stream based on the set of images;
extracting metadata from the image stream;
accessing biometric data generated by a sensor device;
generating a request that includes the metadata and the biometric data to a networked computer;
receiving, from the networked computer, a presentation of the image stream that includes an object rendered at a position within the image stream based on at least the metadata and the biometric data;
causing the display matrix to display the presentation of the image stream that includes the object rendered at the position within the image stream upon the hemispherical element;
detecting biometric feedback; and
presenting the object within the presentation of the image stream based on the biometric feedback.

9. The method of claim 8 , further comprising:
producing eye tracking video of the eye of the user via an inwardly-facing camera mounted on the support structure, wherein the eye tracking video is processed to track pupil position, and wherein the generating the presentation of the image stream is based on the metadata and the eye tracking video.

10. The method of claim 9 , wherein a microcomputer processes the eye tracking video to detect eye comers using oriented gradient features and averages the positions of the eye comers to obtain an x,y location for the eye.

11. The apparatus of claim 9 , wherein a microcomputer processes the eye tracking video to calculate a two-dimensional displacement between the center of the eye and the center of the pupil.

12. The apparatus of claim 9 , wherein a microcomputer processes the eye tracking video to calculate a two-dimensional displacement between the center of the eye and the center of the pupil, and wherein the microcomputer converts the two-dimensional displacement to a three-dimensional gaze vector for the eye.

13. The apparatus of claim 9 , wherein the pupil position is used to alter the position of the object rendered within the image stream.

14. The apparatus of claim 8 , wherein a microcomputer performs a calibration operation by projecting a sequence of objects on a display mounted on the support structure and collecting viewed object acknowledgements from a user.

15. A non-transitory computer readable storage medium storing therein instructions that, when executed by a processor, cause an apparatus for mounting on the head of a user to perform operations comprising:
capturing a set of images by a camera mounted on a support structure of a head-mounted apparatus, the head-mounted apparatus comprising a display matrix, and a plurality of optical lenses retained by the support structure and disposed between the display matrix and an eye of a user;
generating an image stream based on the set of images;
extracting metadata from the image stream;
accessing biometric data generated by a sensor device;
generating a request that includes the metadata and the biometric data to a networked computer;
receiving, from the networked computer, a presentation of the image stream that includes an object rendered at a position within the image stream based on at least the metadata and the biometric data;
causing the display matrix to display the presentation of the image stream that includes the object rendered at the position within the image stream upon the hemispherical element;
detecting biometric feedback; and
presenting the object within the presentation of the image stream based on the biometric feedback.

16. The non-transitory computer readable storage medium of claim 15 , further comprising:
producing eye tracking video of the eye of the user via an inwardly-facing camera mounted on the support structure, wherein the eye tracking video is processed to track pupil position, and wherein the generating the presentation of the image stream is based on the metadata and the eye tracking video.

17. The non-transitory computer readable storage medium of claim 16 , wherein a microcomputer processes the eye tracking video to detect eye comers using oriented gradient features and averages the positions of the eye comers to obtain an x,y location for the eye.

18. The non-transitory computer readable storage medium of claim 16 , wherein a microcomputer processes the eye tracking video to calculate a two-dimensional displacement between the center of the eye and the center of the pupil.

Assignees

• Snap Inc

Inventors

• Miller Erick
• Rodriguez Jonathan

Classifications

• G06T7/73

  using feature-based methods · CPC title

• H04N9/3173 Primary

  wherein the projection device is specially adapted for enhanced portability · CPC title

• G02B27/0176

  characterised by mechanical features · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06T7/60

  Analysis of geometric attributes · CPC title