Automatic generation of virtual materials from real-world materials

Patent metadata

Field	Value
Publication number	US-9779512-B2
Application number	US-201514609312-A
Country	US
Kind code	B2
Filing date	Jan 29, 2015
Priority date	Jan 29, 2015
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

Methods for automatically generating a texture exemplar that may be used for rendering virtual objects that appear to be made from the texture exemplar are described. In some embodiments, a head-mounted display device (HMD) may identify a real-world object within an environment, acquire a three-dimensional model of the real-world object, determine a portion of the real-world object from which a texture exemplar is to be generated, capture one or more images of the portion of the real-world object, determine an orientation of the real-world object, and generate the texture exemplar using the one or more images, the three-dimensional model, and the orientation of the real-world object. The HMD may then render and display images of a virtual object such that the virtual object appears to be made from a virtual material associated with the texture exemplar.

First claim

What is claimed is:

1. An electronic device for generating and controlling virtual objects within an augmented reality environment, comprising:
one or more processors configured to acquire one or more images of a portion of a real-world object and acquire a three-dimensional model of the real-world object, the one or more images include a color image and a depth image, the one or more processors configured to update at least a portion of the color image with interpolated pixel values derived from pixel values within the color image based on depth information from the depth image, the one or more processors configured to identify one or more specular reflections within the one or more images and generate a texture exemplar using the one or more images and the three-dimensional model of the real-world object, the one or more processors configured to generate the texture exemplar using the color image and the depth image such that the one or more specular reflections are substantially removed from the texture exemplar and such that distortions caused by a curved surface of the real-world object are corrected, the one or more processors configured to render images of a virtual object using the texture exemplar such that the virtual object appears to be at least partially covered with a virtual material corresponding with the texture exemplar, the one or more processors configured to generate the texture exemplar and render the images of the virtual object in real-time; and
a display in communication with the one or more processors, the display configured to display the rendered images of the virtual object.

2. The electronic device of claim 1 , wherein:
the one or more processors configured to identify one or more pixel values within the color image that are not part of the real-world object using the depth information from the depth image and update the one or more pixel values within the color image that are not part of the real-world object with the interpolated pixel values derived from other pixel values within the color image that are part of the real-world object.

3. The electronic device of claim 1 , wherein:
the one or more processors configured to render the images of the virtual object using texture synthesis techniques.

4. The electronic device of claim 1 , wherein:
the one or more processors configured to automatically generate the texture exemplar and render the images of the virtual object using the texture exemplar during run-time of an application;
the electronic device comprises a head-mounted display device; and
the display comprises a see-through display.

5. The electronic device of claim 4 , wherein:
the one or more processors configured to cause a virtual camera viewfinder to be displayed using the see-through display and acquire a first image of the one or more images corresponding with the virtual camera viewfinder.

6. The electronic device of claim 1 , wherein:
the one or more processors configured to determine an orientation of the real-world object and generating the texture exemplar based on the orientation of the real-world object.

7. The electronic device of claim 1 , wherein:
the texture exemplar comprises a two-dimensional texture image that depicts the virtual material.

8. A method for generating and controlling virtual objects within an augmented reality environment using a mobile device, comprising:
capturing one or more images of a portion of a real-world object using the mobile device, the one or more images include a color image and a depth image;
acquiring a three-dimensional model of the real-world object;
updating at least a portion of the color image with interpolated pixel values derived from pixel values within the color image based on depth information from the depth image;
identifying one or more specular reflections within the one or more images;
generating a texture exemplar using the one or more images and the three-dimensional model of the real-world object, the texture exemplar is generated such that the one or more specular reflections are substantially removed from the texture exemplar and such that distortions caused by a curved surface of the real-world object are corrected; and
rendering and displaying images of a virtual object using the mobile device such that the virtual object appears to be at least partially covered with a virtual material corresponding with the texture exemplar, the generating the texture exemplar and the rendering and displaying images of the virtual object are performed in real-time by the mobile device.

9. The method of claim 8 , wherein:
the generating the texture exemplar includes unwarping the one or more images using the three-dimensional model of the real-world object such that distortions caused by the curved surface of the real-world object are corrected; and
the generating the texture exemplar and the rendering and displaying images of the virtual object are performed during run-time of an application running on the mobile device.

10. The method of claim 8 , further comprising:
identifying one or more pixel values within the color image that are not part of the real-world object using the depth information from the depth image; and
updating the one or more pixel values within the color image that are not part of the real-world object with the interpolated pixel values derived from other pixel values within the color image that are part of the real-world object.

11. The method of claim 8 , wherein:
the rendering and displaying images of the virtual object includes generating the images using texture synthesis techniques.

12. The method of claim 8 , wherein:
the mobile device comprises a head-mounted display device.

13. The method of claim 12 , further comprising:
projecting a virtual camera viewfinder using the head-mounted display device, the capturing one or more images of a portion of the real-world object includes capturing a first image of the one or more images corresponding with the virtual camera viewfinder.

14. The method of claim 8 , further comprising:
determining an orientation of the real-world object, the generating a texture exemplar includes generating the texture exemplar based on the orientation of the real-world object.

15. The method of claim 8 , wherein:
the texture exemplar comprises a two-dimensional texture image that depicts the virtual material.

16. The method of claim 8 , wherein:
the identifying one or more specular reflections includes capturing two or more images of the portion of the real-world object from two or more different angles, aligning the two or more images with each other, and identifying one or more pixels within the two or more images that do not have corresponding pixel values within a threshold pixel value range.

17. The method of claim 8 , further comprising:
determining a three-dimensional shape for the real-world object using surface reconstruction techniques.

18. One or more storage devices containing processor readable code for programming one or more processors to perform a method for generating and controlling virtual objects within an augmented reality environment using a head-mounted display device, the processor readable code comprising:
processor readable code configured to capture one or more images of a portion of a real-world object using the head-mounted display device, the one or more images include a color image and a depth image;
processor readable code configured to update at least a portion of the color image with interpolated pixel values derived from pixel values

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Tomlin Arthur C
• Sylvan Roger Sebastian-Kevin
• Kroymann Dan
• Brown Cameron G
• Fajt Nicholas Gervase

Classifications

• G06T15/04 Primary

  Texture mapping · CPC title

• G06T7/40 Primary

  Analysis of texture (depth or shape recovery from texture G06T7/529) · CPC title

• G02B2027/0138

  comprising image capture systems, e.g. camera · CPC title

• G06T5/006

  Physics · mapped topic

• G02B2027/0178

  Eyeglass type (eyeglass details G02C) · CPC title