Headware with computer and optical element for use therewith and systems utilizing same

Patent metadata

Field	Value
Publication number	US-11099643-B1
Application number	US-201916667489-A
Country	US
Kind code	B1
Filing date	Oct 29, 2019
Priority date	May 11, 2011
Publication date	Aug 24, 2021
Grant date	Aug 24, 2021

Abstract

An apparatus for mounting on a head including a frame, A face-wearable near-ocular optics and a micro-display for displaying data in front of the eyes is provided. A computing device is coupled to the micro-display. At least one sensor is coupled to the computing device for receiving biometric human information.

First claim

What is claimed is:

1. An apparatus comprising:
a support structure for mounting on the head of a user;
a microcomputer carried by the support structure;
an outwardly-facing camera mounted on the support structure; and
a transceiver coupled with the microcomputer, wherein the microcomputer executes a first part of a procedure to extract data from images captured by the outwardly-facing camera, and sends a second part of the procedure to a mobile computer via the transceiver for execution of the second part of the procedure on the mobile computer, the result of the second part of the procedure made available to the microcomputer, wherein the first part of the procedure is executed by the microcomputer simultaneously as the second part of the procedure is executed on the mobile computer.

2. The apparatus of claim 1 , further comprising:
an inwardly-facing camera mounted on the support structure, the inwardly-facing camera producing eye tracking video of an eye that is processed by the microcomputer, wherein the microcomputer processes the eye tracking video to track pupil position.

3. The apparatus of claim 2 , wherein the microcomputer processes the eye tracking video to detect eye corners using oriented gradient features and averages the positions of the eye corners to obtain an x,y location for the eye.

4. The apparatus of claim 2 , wherein the microcomputer processes the eye tracking video to calculate a two-dimensional displacement between the center of the eye and the center of the pupil.

5. The apparatus of claim 4 , wherein the microcomputer converts the two-dimensional displacement to a three-dimensional gaze vector for the eye.

6. The apparatus of claim 5 , wherein the microcomputer applies an affine transformation in three-dimensions to convert the two-dimensional displacement into the three-dimensional gaze vector for the eye.

7. The apparatus of claim 5 , wherein the microcomputer computes a three-dimensional point in space where two gaze rays intersect to establish a z coordinate vergence point.

8. The apparatus of claim 2 , wherein the pupil position is used to alter a position of data projected on a display mounted on the support structure.

9. The apparatus of claim 1 , wherein the microcomputer performs a calibration operation by projecting a sequence of objects on a display mounted on the support structure and collecting viewed object acknowledgements from a user.

10. The apparatus of claim 1 , further comprising:
a sensor fusion controller to combine data from a plurality of sensors; and
wherein the plurality of sensors comprise at least one of:

an accelerometer, a gyroscope, a magnetometer, an altimeter, a clock, or a thermometer.

11. The apparatus of claim 1 , wherein the microcomputer processes the images captured by the outwardly-facing camera by overlaying virtual objects onto the images.

12. The apparatus of claim 1 , wherein the apparatus is connected with a local network comprising the mobile computer.

13. A method comprising:
capturing images using an outwardly-facing camera of an apparatus for mounting on the head of a user;
generating, by a microcomputer of the apparatus, an image stream from the captured images of the outwardly-facing camera;
executing, by the microcomputer of the apparatus, a first part of a procedure to extract data from images captured by the outwardly-facing camera of the apparatus;
determining, by the microcomputer of the apparatus, a mobile computer to execute a second part of the procedure;
transmitting, by the microcomputer of the apparatus, the second part of the procedure to the mobile computer for execution of the second part of the procedure on the mobile computer; and
receiving, by the microcomputer of the apparatus, a result of the execution of the second part of the procedure from the mobile computer;
wherein the first part of the procedure is executed by the microcomputer simultaneously as the second part of the procedure is executed on the mobile computer.

14. The method of claim 13 , further comprising:
producing, using an inwardly-facing camera of the apparatus, an eye tracking video of an eye; and
processing, by the microcomputer of the apparatus, the eye tracking video to track pupil position.

15. The method of claim 13 , further comprising:
causing, by the microcomputer of the apparatus, the image stream to be displayed on a display of the apparatus as displayed data; and
altering, by the microcomputer of the apparatus, the displayed data to cause a rendered object of the displayed data to move according to an eye movement of the user, based on the tracked pupil position.

16. The method of claim 13 , further comprising:
sending the displayed data to a networked computer via a communication network.

17. The method of claim 13 , wherein the result of the procedure is made available to the mobile computer.

18. The method of claim 13 , further comprising:
using the pupil position to alter a position of data projected on a display of the apparatus.

19. A computer readable storage medium storing therein instructions that, when executed by a processor, cause an apparatus for mounting on the head of a user to perform operations comprising:
capturing images using an outwardly-facing camera of the apparatus;
generating an image stream from the captured images of the outwardly-facing camera;
executing a first part of a procedure to extract data from images captured by the outwardly-facing camera of the apparatus;
determining a mobile computer to execute a second part of the procedure;
transmitting the second part of the procedure to the mobile computer for execution of the second part of the procedure on the mobile computer; and
receiving a result of the execution of the second part of the procedure from the mobile computer;
wherein the first part of the procedure is executed by the microcomputer simultaneously as the second part of the procedure is executed on the mobile computer.

Assignees

• Snap Inc

Inventors

• Miller Erick
• Rodriguez Jonathan

Classifications

• H04N9/3173 Primary

  wherein the projection device is specially adapted for enhanced portability · CPC title

• G06V40/19

  Sensors therefor · CPC title

• G06V10/803

  of input or preprocessed data · CPC title

• G06F3/013 Primary

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F18/251

  of input or preprocessed data · CPC title