Systems and methods for dynamic detection and presentation of obscured real-world objects in augmented or mixed reality virtual content

Patent metadata

Field	Value
Publication number	US-12217329-B2
Application number	US-202117497519-A
Country	US
Kind code	B2
Filing date	Oct 8, 2021
Priority date	Oct 8, 2021
Publication date	Feb 4, 2025
Grant date	Feb 4, 2025

Abstract

A method for dynamic detection and presentation of obscured real-world objects in augmented or mixed reality virtual content may include an object of interest computer program executed by an electronic device: (1) receiving data for a virtual or augmented reality object to display on an image of a physical environment on a display of the electronic device; (2) receiving data for a physical object in the physical environment; (3) determining that the physical object is obscured by the virtual or augmented reality object; (4) generating a representation of the physical object; and (5) providing the representation of the physical object to a virtual reality or augmented reality computer program executed by the electronic device, wherein the virtual reality or augmented reality computer program is configured to display the physical object over the virtual or augmented reality object.

First claim

What is claimed is:

1. A method, comprising:
receiving, by an object of interest computer program executed by an electronic device, a location for a virtual or augmented reality object to display on an augmented reality view of a physical environment by an augmented reality computer program executed by the electronic device, wherein the augmented reality view comprises a physical object that is fixed in the physical environment;
generating, by the object of interest computer program, a graphical representation of the virtual or augmented reality object at the location for the virtual or augmented reality object;
detecting, by the object of interest computer program, the physical object from image data or sensor data received from the electronic device;
determining, by the object of interest computer program, a location for the physical object in the physical environment from the image data or the sensor data;
determining, by the object of interest computer program by using the location for the virtual or augmented reality object and the location for the physical object, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object on the augmented reality view of the physical environment displayed on the electronic device;
generating, by the object of interest computer program, a virtual representation of the physical object; and
providing, by the object of interest computer program, the virtual representation of the physical object to the augmented reality computer program, wherein the augmented reality computer program is configured to display the virtual representation of the physical object over the graphical representation of the virtual or augmented reality object in the augmented reality view of the physical environment at the location for the physical object.

2. The method of claim 1 , wherein the object of interest computer program receives the location for the virtual or augmented reality object from the augmented reality computer program.

3. The method of claim 1 , further comprising:
determining, by the object of interest computer program, that the physical object is an object of interest.

4. The method of claim 1 , wherein the physical object comprises a dangerous, hazardous, or harmful object.

5. The method of claim 1 , wherein the step of determining, by the object of interest computer program, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object comprises:
determining that the location of the graphical representation of the virtual or augmented reality object is between the electronic device and the physical object.

6. The method of claim 5 , wherein the object of interest computer program determines a location of the physical object using a ray cast query.

7. The method of claim 5 , wherein the object of interest computer program determines a location of the physical object using LiDAR.

8. The method of claim 1 , wherein the virtual representation of the physical object comprises an image of the physical object.

9. A method, comprising:
receiving, by an object of interest computer program executed by an electronic device, a location for a virtual or augmented reality object to display on an augmented reality view of a physical environment by an augmented reality computer program executed by the electronic device, wherein the augmented reality view comprises a physical object that is fixed in the physical environment;
generating, by the object of interest computer program, a graphical representation of the virtual or augmented reality object at the location for the virtual or augmented reality object;
detecting, by the object of interest computer program, the physical object from image data or sensor data received from the electronic device;
determining, by the object of interest computer program, a location for the physical object in the physical environment from the image data or the sensor data;
determining, by the object of interest computer program by using the location for the virtual or augmented reality object and the location for the physical object, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object on the augmented reality view of the physical environment displayed on the electronic device;
detecting, by the object of interest computer program, a boundary of the physical object; and
providing, by the object of interest computer program, the boundary of the physical object to the augmented reality computer program, wherein the augmented reality computer program is configured to remove an area within the boundary from the graphical representation of the virtual or augmented reality object, wherein the physical object is visible in the area of the graphical representation of the virtual or augmented reality object.

10. The method of claim 9 , wherein the object of interest computer program receives the location for the virtual or augmented reality object from the augmented reality computer program.

11. The method of claim 9 , further comprising:
determining, by the object of interest computer program, that the physical object comprises a dangerous, hazardous, or harmful object.

12. The method of claim 11 , wherein the step of determining, by the object of interest computer program, that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object comprises:
determining that the location of the graphical representation of the virtual or augmented reality object is between the electronic device and the physical object.

13. The method of claim 12 , wherein the object of interest computer program determines a location of the physical object using a ray cast query.

14. The method of claim 12 , wherein the object of interest computer program determines a location of the physical object using LiDAR.

15. An electronic device, comprising:
a display;
an object of interest computer program; and
a computer processor;
wherein, when executed by the computer processor, the object of interest computer program causes the computer processor to:
receive a location for a virtual or augmented reality object to display on an augmented reality view of a physical environment by an augmented reality computer program executed by the electronic device, wherein the augmented reality view comprises a physical object that is fixed in the physical environment;
generate a graphical representation of the virtual or augmented reality object at the location for the virtual or augmented reality object;
detecting the physical object from image data or sensor data received from the electronic device;
determine a location for the physical object in the physical environment from the image data or the sensor data;
determine that the physical object should not be obscured by identifying a type of the physical object in an object of interest database comprising a plurality of types of physical objects;
determine by using the location for the virtual or augmented reality object and the location for the physical object that the physical object is at least partially obscured by the graphical representation of the virtual or augmented reality object on the augmented reality view of the physical environment displayed on the electronic device;
generate a virtual representation of the physical object; and
provide the virtual representation of the physical object to the augmented reality computer program, wherein the augmente

Assignees

• Jpmorgan Chase Bank Na

Inventors

• Pistoia Marco
• Moriarty Bill
• Hu Shaohan
• Kalsi Hargun
• Arantes Aniella

Classifications

• G01S17/89

  for mapping or imaging · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title