Virtually visualizing energy

Patent metadata

Field	Value
Publication number	US-9922463-B2
Application number	US-201514821632-A
Country	US
Kind code	B2
Filing date	Aug 7, 2015
Priority date	Aug 7, 2015
Publication date	Mar 20, 2018
Grant date	Mar 20, 2018

Abstract

The techniques describe herein use sensor(s) to scan a real-world environment and obtain data associated with geometry of the real-world environment that affects how energy propagates (e.g., locations of spatial objects in a room). The sensor(s) also detect energy (e.g., sound) in the real-world environment, from which a location of a source of the energy can be determined. The techniques combine the geometry data and the energy data to determine how the detected energy propagates from the location of the source through the real-world environment. The techniques can then cause a representation of the propagating energy to be displayed, to a user, as virtual content via a mixed reality device. Accordingly, a user is able to see energy that is otherwise invisible.

First claim

What is claimed is:

1. A system comprising:
one or more processors;
memory; and
one or more modules stored in the memory and executable by the one or more processors to perform operations comprising:
receiving first data associated with real objects and boundaries that affect the propagation of energy within an environment in which a user is present;
computing, based at least in part on the received first data, geometry-based properties, wherein the geometry-based properties comprise a reflection map to model how the energy is redirected in response to encountering a real or a virtual object;
receiving second data associated with energy in the environment;
locating, based at least in part on the received second data, a source of the energy within the environment;
using the geometry-based properties to compute a representation of how the energy propagates from a location of the source of the energy through the environment; and
causing the representation of how the energy propagates from the location of the source of the energy through the environment to be rendered on a display.

2. The system as claim 1 recites, wherein the display is part of, or coupled to a mixed reality device, and the display comprises a transparent display.

3. The system as claim 1 recites, wherein the representation comprises a set of spheres distributed throughout the environment and a radius of an individual sphere is determined based on an intensity of the energy at a particular three-dimensional location within the environment.

4. The system as claim 3 recites, wherein a first sphere in the set of spheres has a first radius that is greater than a second radius of a second sphere in the set of spheres, the first sphere being closer to the source of the energy than the second sphere.

5. The system as claim 1 recites, wherein the representation comprises a set of spheres distributed throughout the environment and a color of an individual sphere is determined based on the energy at a particular three-dimensional location within the environment belonging to a particular frequency band of a plurality of different frequency bands.

6. The system as claim 1 recites, wherein the geometry-based properties further comprise at least one of:
a loudness map for sound propagation; or
a reverberation map to model how repeated energy propagates.

7. The system as claim 1 recites, the operations further comprising:
receiving third data associated with another energy in the environment;
locating, based at least in part on the received third data, another source of the other energy within the environment;
using the geometry-based properties to compute another representation of how the other energy propagates from a location of the other source of the other energy through the environment; and
causing the other representation of how the other energy propagates from the location of the other source of the other energy through the environment to be rendered on the display simultaneously with the representation of how the energy propagates from the location of the source of the energy through the environment.

8. The system as claim 1 recites, the operations further comprising:
determining a location and an orientation of a mixed reality device associated with the user in the environment;
determining a viewing perspective of the user based at least in part on the location and the orientation of the mixed reality device; and
using the geometry-based properties to re-compute the representation of how the energy propagates from the location of the source of the energy through the environment based at least in part on the viewing perspective of the user.

9. The system as claim 1 recites, wherein the energy is associated with one of:
sound;
a Wi-Fi signal;
heat;
light;
fluid dynamics;
an electro-magnetic field;
radiation; or
an air current.

10. A method comprising:
receiving first data associated with real objects and boundaries that affect propagation of energy within an environment in which a user is present;
computing, by one or more processors and based at least in part on the received first data, geometry-based properties, wherein the geometry-based properties comprise a loudness map for sound propagation;
receiving second data associated with energy in the environment;
locating, based at least in part on the received second data, a source of the energy within the environment;
using the geometry-based properties to compute a representation of how the energy propagates from a location of the source of the energy through the environment; and
causing the representation of how the energy propagates from the location of the source of the energy through the environment to be rendered on a display.

11. The method as claim 10 recites, wherein the display is part of, or coupled to a mixed reality device, and the display comprises a transparent display.

12. The method as claim 10 recites, wherein the representation comprises a set of spheres distributed throughout the environment and a radius of an individual sphere is determined based on an intensity of the energy at a particular three-dimensional location within the environment.

13. The method as claim 12 recites, wherein a first sphere in the set of spheres has a first radius that is greater than a second radius of a second sphere in the set of spheres, the first sphere being closer to the source of the energy than the second sphere.

14. The method as claim 10 recites, wherein the representation comprises a set of spheres distributed throughout the environment and a color of an individual sphere is determined based on the energy at a particular three-dimensional location within the environment belonging to a particular frequency band of a plurality of different frequency bands.

15. The method as claim 10 recites, wherein the geometry-based properties further comprise at least one of:
a reflection map to model how the energy is redirected in response to encountering a real or a virtual object; or
a reverberation map to model how repeated energy propagates.

16. The method as claim 10 recites, further comprising:
determining a location and an orientation of a mixed reality device associated with the user in the environment;
determining a viewing perspective of the user based at least in part on the location and the orientation of the mixed reality device; and
using the geometry-based properties to re-compute the representation of how the energy propagates from the location of the source of the energy through the environment based at least in part on the viewing perspective of the user.

17. A system configured to communicate with a mixed reality device located within a real-world scene, the system comprising:
one or more sensors;
one or more processors; and
memory storing instructions that, when executed on the one or more processors, cause the system to perform operations comprising:
scanning, using the one or more sensors, the real-world scene to obtain first data associated with real objects that affect the propagation of energy within the real-world scene in which a user is present;
computing, based at least in part on the obtained first data, geometry-based properties of the real-world scene, wherein the geometry-based properties comprise a reverberation map to model how repeated energy propagates;
detecting, using the one or more sensors, second data associated with energy in the real-world scene;
locating, based at least in part on the detected second data, a source of the energy within the real-world scen

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Lanier Jaron
• Rathinavel Kishore
• Raghuvanshi Nikunj

Classifications

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G09B23/06

  for physics · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T11/20

  Drawing from basic elements · CPC title