Optical see through (OST) head mounted display (HMD) system and method for precise alignment of virtual objects with outwardly viewed objects

Patent metadata

Field	Value
Publication number	US-12399559-B2
Application number	US-202418783139-A
Country	US
Kind code	B2
Filing date	Jul 24, 2024
Priority date	Apr 27, 2021
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

A method for irradiating an image in an optical see-through (OST) head mounted display (HMD) for viewing through, the OST HMD by a user's eye, an object having at least one of known orientation and position and orientation (O/P&O), associated with a first reference frame, the method comprising: generating and irradiating said image for appearing to said user superimposed in an aligned manner to said object, according to predetermined information, eyeball feature position data, and said O/P&O; said predetermined information relates correction data with a plurality of different respective position data values of at least one eyeball feature position of said eye; said predetermined information further includes display corrections of said electro-optical display module with respect to said position data values of said at least one eyeball feature position, with respect to a second reference frame; and said O/P&O is between said second reference and said first reference frame.

First claim

The invention claimed is:

1. An optical see-through (OST) head mounted display (HMD) system, for viewing an image aligned with an object associated with a first reference frame, the OST HMD system comprising:
an electro-optical display device including:
at least one partially reflective partially transmissive optical element for at least one of viewing therethrough said object, and viewing said image; and
at least one electro-optical projector configured to irradiate said image for viewing by at least one eye of a user who wears said OST HMD, said electro-optical display device being associated with a second reference frame; and

a processor configured to be coupled with said electro-optical display device, said processor configured to generate said image, according to predetermined information, eyeball feature position data, and at least one of an orientation, and a position and orientation, so that said image appears to said user in an aligned manner with respect to said object;
wherein said eyeball feature position data is associated with at least one eyeball feature position of said at least one eye, with respect to said second reference frame; wherein said at least one of orientation, and position and orientation is of said second reference frame with respect to said first reference frame; and
wherein said predetermined information relates correction data of said OST HMD to a plurality of different respective position data values of said at least one eyeball feature position;
wherein said predetermined information further includes display corrections of said electro-optical display device with respect to said position data values of said at least one eyeball feature position,
wherein said predetermined information further includes see-through corrections of said at least one partially reflective partially transmissive optical element,
wherein said predetermined information is attained via a calibration procedure with respect to said plurality of different respective position data,
wherein said eyeball feature position data is retrieved from at least one of: (1) an eye position determinator (2) said processor receiving said eyeball feature position data from at least one of: a user, a memory device and a communication module (3) via a user calibration stage.

2. The system according to claim 1 , wherein said eye position determinator is configured to determine said at least one eyeball feature position of said at least one eye with respect to said second reference frame, and to generate corresponding said eyeball feature position data.

3. The system according to claim 2 , wherein said eye position determinator includes a camera configured to acquire at least one image of said at least one eye.

4. The system according to claim 2 , wherein said eye position determinator and said electro-optical display device are mechanically coupled.

5. The system according to claim 2 , wherein said eye position determinator and said electro-optical display device are configured as an on-axis optical arrangement.

6. The system according to claim 2 , wherein said determinator is external to said system.

7. The system according to claim 2 , wherein said eye position determinator is configured to output said eyeball feature position data in either one of: continuously, and from time to time.

8. The system according to claim 1 , further comprising a tracking system configured to determine said at least one of an orientation, and a position and orientation, of said second reference frame with respect to said first reference frame.

9. The system according to claim 1 , further including a memory device for storing said predetermined information, wherein said processor is configured to read said predetermined information from said memory device.

10. The system according to claim 1 , wherein in case said at least one eyeball feature position of said at least one eye does not precisely correspond with one of said plurality of different respective position data values of said predetermined information, said processor is configured to either one of: (1) derive from said correction data closest-matching correction data associated with at least one closest matching position, or (2) derive from said correction data interpolated correction data by interpolating from correction data associated with at least one closest matching position.

11. The system according to claim 1 , wherein said OST HMD is used for surgical applications where said object is an anatomical feature.

12. The system according to claim 1 , wherein said eyeball feature position data is acquired via said user calibration stage by projecting a user calibration image to said user to be aligned with a real-world object.

13. The system according to claim 1 , wherein said predetermined information is stored in a look-up table.

14. A method for irradiating an image in an optical see-through (OST) head mounted display (HMD) for viewing through the OST HMD an object having at least one of a known orientation and a known position and orientation, with respect to a first reference frame, by at least one eye of a user, the method comprising:
a calibration procedure for acquiring predetermined information with respect to a plurality of different respective eyeball feature position data;
generating and irradiating said image for viewing by said user, via en electro-optical display device, such that said image appears to said user superimposed in an aligned manner with respect to said object, according to said predetermined information, said eyeball feature position data, and at least one of an orientation, and a position and orientation;
wherein said predetermined information relates correction data of said OST HMD with a plurality of different respective position data values of at least one eyeball feature position of said at least one eye;
wherein said predetermined information further includes display corrections of said electro-optical display device with respect to said position data values of said at least one eyeball feature position,
wherein said predetermined information further includes see-through corrections of said at least one partially reflective partially transmissive optical element,
wherein said eyeball feature position data is associated with at least one eyeball feature position of said at least one eye, with respect to a second reference frame; and
wherein said at least one of orientation and position and orientation is of said second
reference frame with respect to said first reference frame;
wherein said eyeball feature position data is retrieved from at least one of: (1) an eye position determinator (2) said processor receiving said eyeball feature position data from at least one of: a user, a memory device and a communication module (3) via a user calibration stage.

15. The method according to claim 14 , further comprising generating said eyeball feature position data, by determining corresponding said at least one eyeball feature position of said at least one eye with respect to said second reference frame.

16. The method according to claim 15 , further comprising deriving from said correction data either one of: (1) closest-matching correction data in case determined said at least one eyeball feature position of said at least one eye does not precisely correspond with one of said plurality of different respective position data values of said predetermined information, or (2) interpolated correction data by interpolating from correction data associated with at least one closest matching position.

17. The method accordi

Assignees

• Elbit Systems Ltd

Inventors

• Ben-Yishai Rani
• Benesh Gil

Classifications

• G02B27/0172

  characterised by optical features · CPC title

• H04N13/327

  Calibration thereof · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• A61B2034/258

  providing specific settings for specific users · CPC title

• A61B2017/00216

  with eye tracking or head position tracking control · CPC title