Method and apparatus for the interaction of virtual tools and geo-registered tools

Patent metadata

Field	Value
Publication number	US-11207133-B1
Application number	US-201916563985-A
Country	US
Kind code	B1
Filing date	Sep 9, 2019
Priority date	Sep 10, 2018
Publication date	Dec 28, 2021
Grant date	Dec 28, 2021

Abstract

Virtual objects can be affixed to geo-registered tools (e.g., platform, knife, etc.) that interact with items (e.g., anatomic structures) within 3D volumes. For example, a 27-foot virtual wand can be affixed to a 1-foot long geo-registered wand and the resulting multi-part wand can be moved another two feet within a 3D volume in response to movement of the geo-registered wand to reach an object 30 feet away in the 3D volume. Various other interaction techniques are disclosed. The techniques may be used in a variety of applications possibly including, but not limited to, medicine, gaming, education, carpentry, manufacturing, or other industries.

First claim

The invention claimed is:

1. A method comprising:
performing geo-registration of a virtual object to a first tangible object:
wherein the virtual object has location coordinates in physical space;
wherein a change in the first tangible object's position causes a corresponding change in the virtual object's position; and
wherein a change in the first tangible object's orientation causes a corresponding change in the virtual object's orientation;

performing geo-registration of a virtual tool to a second tangible object:
wherein the second tangible object is held in a user's hand;
wherein the virtual tool has location coordinates in physical space;
wherein a change in the second tangible object's position causes a corresponding change in the virtual tool's position; and
wherein a change in the second tangible object's orientation causes a corresponding change in the virtual tool's orientation;

tracking the first tangible object's position and orientation and the second tangible object's position and orientation;
determining a change in the virtual object's position and orientation, which corresponds to a change in the first tangible object's position and orientation;
determining a change in the virtual tool's position and orientation, which correspond to a change in the second tangible object's position and orientation;
causing the virtual object to be manipulated by the virtual tool:
wherein the manipulation occurs in response to placement of the virtual tool into the virtual object; and
wherein the manipulation comprises an alteration of the virtual object's three-dimensional structure;
assigning a virtual task to the user, in which the user manipulates the virtual object using the virtual tool to perform the assigned virtual task, wherein assigning the virtual task to the user comprises assigning a medical procedure; and

displaying the manipulated virtual object to the user wearing an extended reality head display unit.

2. The method of claim 1 further comprising:
assigning a set of coordinates on the virtual object designated as a spot; and
performing a programmed response when the spot is touched by the virtual tool.

3. The method of claim 2 wherein the programmed response is an information box.

4. The method of claim 1 wherein the virtual tool comprises a grabbing tool.

5. The method of claim 1 wherein the virtual tool comprises a pointing tool.

6. The method of claim 1 wherein the virtual tool comprises a cutting tool.

7. The method of claim 1 wherein the virtual tool comprises a writing tool.

8. The method of claim 1 comprising detecting by using an eye tracking system a location in the virtual object upon which the user's eyes are focused and providing an indication of the location in the virtual object upon which the user's eyes are focused to a second user who is wearing a second extended reality head display unit and looking at the virtual object.

9. The method of claim 1 comprising prompting performance of an additional task selected from the group consisting of: treating a stroke; treating an aneurysm; emplacing a virtual stent; performing a chemoembolization; and conducting a lung biopsy.

10. The method of claim 1 comprising scoring performance of the virtual task based on manipulation of the item virtual object.

11. The method of claim 1 comprising scoring performance based on correct identification of medical terminology for differing parts of anatomy and associated bodily functions.

12. The method of claim 1 comprising scoring performance based on accuracy and completeness of a diagnosis.

13. The method of claim 1 comprising scoring performance based on accuracy and time to complete a surgical operation.

14. The method of claim 1 comprising scoring performance based on accuracy and time to complete an interventional operation.

15. An apparatus comprising:
a first tangible object;
a second tangible object;
an extended reality head display unit; and
an image processor comprising a program stored on a computer-readable non-transitory media, the program comprising instructions for:
performing geo-registration of a virtual object to a first tangible object:
wherein the virtual object has location coordinates in physical space;
wherein a change in the first tangible object's position causes a corresponding change in the virtual object's position; and
wherein a change in the first tangible object's orientation causes a corresponding change in the virtual object's orientation;

performing geo-registration of a virtual tool to a second tangible object:
wherein the second tangible object is held in a user's hand;
wherein the virtual tool has location coordinates in physical space;
wherein a change in the second tangible object's position causes a corresponding change in the virtual tool's position; and
wherein a change in the second tangible object's orientation causes a corresponding change in the virtual tool's orientation;

tracking the first tangible object's position and orientation and the second tangible object's position and orientation;
determining a change in the virtual object's position and orientation, which corresponds to a change in the first tangible object's position and orientation;
determining a change in the virtual tool's position and orientation, which correspond to a change in the second tangible object's position and orientation;
causing the virtual object to be manipulated by the virtual tool:
wherein the manipulation occurs in response to placement of the virtual tool into the virtual object; and
wherein the manipulation comprises an alteration of the virtual object's three-dimensional structure;

assigning a virtual task to the user, in which the user manipulates the virtual object using the virtual tool to perform the assigned virtual task, wherein assigning the virtual task to the user comprises assigning a medical procedure; and
displaying the manipulated virtual object to the user wearing an extended reality head display unit.

16. The apparatus of claim 15 further comprising instructions for:
assigning a set of coordinates on the virtual object designated as a spot; and
performing a programmed response when the spot is touched by the virtual tool.

17. The apparatus of claim 16 wherein the programmed response is an information box.

18. The apparatus of claim 15 wherein the virtual tool comprises a grabbing tool.

19. The apparatus of claim 15 wherein the virtual tool comprises a pointing tool.

20. The apparatus of claim 15 wherein the virtual tool comprises a cutting tool.

21. The apparatus of claim 15 wherein the virtual tool comprises a writing tool.

22. The apparatus of claim 15 comprising an eye tracking system that detects a location in the virtual object upon which the user's eyes are focused and an image generator that provides an indication of the location in the virtual object upon which the user's eyes are focused to a second user who is wearing a second extended reality head display unit and looking at the virtual object.

23. The apparatus of claim 15 comprising an additional task selected from the group consisting of: treating a stroke; treating an aneurysm; emplacing a virtual stent; performing a chemoembolization; and conducting a lung biopsy.

24. The apparatus of claim 15 comprising a processor that calculates a performance score for the virtual task based on manipulation of the item virtual object.

Assignees

• Douglas David Byron
• Douglas Robert Edwin
• Douglas Kathleen Mary

Inventors

• Douglas David Byron
• Douglas Robert Edwin
• Douglas Kathleen Mary

Classifications

• A61B2090/502

  Headgear, e.g. helmet, spectacles · CPC title

• A61B2090/365

  augmented reality, i.e. correlating a live optical image with another image · CPC title

• A61B2034/105

  Modelling of the patient, e.g. for ligaments or bones · CPC title

• A61B2034/104

  Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring · CPC title

• A61B2034/101

  Computer-aided simulation of surgical operations · CPC title