Method and apparatus for selectively integrating sensory content

Patent metadata

Field	Value
Publication number	US-9916681-B2
Application number	US-201514929324-A
Country	US
Kind code	B2
Filing date	Oct 31, 2015
Priority date	Nov 4, 2014
Publication date	Mar 13, 2018
Grant date	Mar 13, 2018

Abstract

To integrate a sensory property such as occlusion, shadowing, reflection, etc. among physical and notional (e.g. virtual/augment) visual or other sensory content, providing an appearance of similar occlusion, shadowing, etc. in both models. A reference position, a physical data model representing physical entities, and a notional data model are created or accessed. A first sensory property from either data model is selected. A second sensory property is determined corresponding with the first sensory property, and notional sensory content is generated from the notional data model with the second sensory property applied thereto. The notional sensory content is outputted to the reference position with a see-through display. Consequently, notional entities may appear occluded by physical entities, physical entities may appear to cast shadows from notional light sources, etc.

First claim

We claim:

1. A method, comprising:
capturing an image representative of a real-world environment;
defining, by a processor, a reference point in the image;
determining a first set of data representative of an object in the image at the reference point;
determining a second set of data representative of a surface of the object;
identifying a first subset of data in the second set of data representative a potential shadow on the surface of the object, wherein the potential shadow indicates a shadow may be located on at least a portion of the surface of the object based on a feature of the second set of data;
identifying a second subset of data in the second set of data representative of a light source in the image based on the second set of data;
determining an anticipated shadow on the surface of the object from the light source, wherein the anticipated shadow is a shadow that may be located on the surface of the object based on an illumination pattern of the light source;
generating notional sensory content to augment the image of the real-world environment at the reference point; and
determining that the potential shadow matches the anticipated shadow; and
in response to the potential shadow matching the anticipated shadow, applying the potential shadow onto the notional sensory content.

2. The method of claim 1 , wherein determining the first set of data comprises:
obtaining, by a depth sensor, a depth measurement of the real-world environment;
communicating the depth measurement to the processor; and
determining, by the processor, the first set of data from the depth measurement.

3. The method of claim 2 , wherein the depth sensor comprises at least one of a depth camera, a stereo camera, a plenoptic camera, a time-of-flight sensor, an ultrasonic sensor, or a millimeter wave sensor.

4. The method of claim 1 , wherein determining the second set of data comprises:
Obtaining, by an image sensor, the image;
communicating the image to the processor; and
determining, by the processor, the second set of data from the image.

5. The method of claim 4 , wherein the image sensor comprises at least one of a color camera, a grayscale camera, a black-and-white camera, a digital camera, or an infrared camera.

6. The method of claim 1 , wherein the light source is explicit in the second set of data.

7. The method of claim 1 , wherein the light source is implicit in the second set of data.

8. The method of claim 1 , wherein the second subset of data is explicit in the first set of data.

9. The method of claim 1 , wherein the second subset of data is implicit in the first set of data.

10. The method of claim 1 , wherein at least a portion of the second set of data is persistent.

11. The method of claim 1 , wherein at least a portion of the first set of data is persistent.

12. The method of claim 1 , wherein the feature of the second set of data is a color feature of the object or a structure feature of the object.

13. The method of claim 1 , further comprising identifying the potential shadow using an algorithm applied to the second set of data.

14. The method of claim 1 , comprising: identifying the light source from at least one of a color feature of the object or a structure feature of the object.

15. The method of claim 1 , comprising identifying the light source by an algorithm applied to the second set of data.

16. A method, comprising:
capturing an image representative of a real-world environment;
defining, by a processor, a viewing position for a user;
determining, by a digital time-of-flight depth camera, a digital depth image of real-world environment;
determining, by the processor, a first set of data representative of an object in real-world environment;
receiving, from a digital color camera, a digital color image of the real-world environment;
determining, by the processor, a second set of data representative of a surface of the object;
identifying a first subset of data in the second set of data representative a potential shadow on the surface of the object, wherein the potential shadow indicates a shadow may be located on at least a portion of the surface of the object based on a feature of the second set of data;
identifying a second subset of data in the second set of data representative of a light source in the image based on the second set of data;
determining an anticipated shadow on the surface of the object from the light source, wherein the anticipated shadow is a shadow that may be located on the surface of the object based on an illumination pattern of the light source;
generating, by the processor, augmented reality graphical content to augment the image of the real-world environment at a point of view;
determining that the potential shadow matches the anticipated shadow; and
in response to the potential shadow matching the anticipated shadow, applying the potential shadow onto the augmented reality graphical content.

17. An apparatus, comprising:
a processor;
a see-through display coupled to the processor;
a reference position establisher comprising executable instructions instantiated on the processor, the reference position establisher being adapted to establish a reference point in an image representative of a real-world environment;
a solids data establisher comprising executable instructions instantiated on the processor, the solids data establisher being adapted to determine a first set of data representative of an object at the reference point;
a surfaces data establisher comprising executable instructions instantiated on the processor, the surfaces data establisher being adapted to determine a second set of data representative of a surface of the object;
a potential shadow identifier comprising executable instructions instantiated on the processor, the potential shadow identifier being adapted to identify a first subset of data in the second set of data representative a potential shadow on the surface of the object, wherein the potential shadow indicates a shadow may be located on at least a portion of the surface of the object based on a feature of the second set of data;
a light source locator comprising executable instructions instantiated on the processor, the light source locator being adapted to identifying a second subset of data in the second set of data representative of a light source in the image based on the second set of data;
an anticipated shadow determiner comprising executable instructions instantiated on the processor, the anticipated shadow determiner being adapted to determine an anticipated shadow on the surface of the object from the light source, wherein the anticipated shadow is a shadow that may be located on the surface of the object based on an illumination pattern of the light source;
a shadow matcher comprising executable instructions instantiated on the processor, the shadow matcher being adapted to determine that the potential shadow matches the anticipated shadow; and
a notional content generator comprising executable instructions instantiated on the processor, the notional content generator being adapted to:
generate notional content; and
in response to the potential shadow matching the anticipated shadow, apply the potential shadow onto the notional content.

18. The apparatus of claim 17 , comprising:
a depth sensor in communication with the processor, the depth sensor being adapted to obtain at a depth measurement of the real-world environment; and
wherein the solids data establisher is adapted to determine the first set of data from the depth m

Assignees

• Atheer Inc

Inventors

• James Greg
• Yang Allen Yang
• Itani Sleiman

Classifications

• H04N23/00

  Cameras or camera modules comprising electronic image sensors; Control thereof · CPC title

• G02B2027/014

  comprising information/image processing systems · CPC title

• G06T15/60 Primary

  Shadow generation · CPC title

• G02B27/0101

  characterised by optical features (G02B27/0172 takes precedence) · CPC title

• G02B27/0172

  characterised by optical features · CPC title