Construction of synthetic augmented reality environment

Patent metadata

Field	Value
Publication number	US-9704295-B2
Application number	US-201314072708-A
Country	US
Kind code	B2
Filing date	Nov 5, 2013
Priority date	Nov 5, 2013
Publication date	Jul 11, 2017
Grant date	Jul 11, 2017

Abstract

Embodiments are disclosed that relate to producing a synthetic environmental model derived from a three dimensional representation of an environment, and rendering images from the model. For example, one disclosed embodiment provides a method including detecting a trigger to build the synthetic environmental model utilizing the three dimensional representation of the environment, and, in response to the trigger, obtaining a set of synthetic image elements for use in constructing the synthetic environmental model. The method further includes fitting one or more elements from the set of synthetic image elements to the three dimensional representation of the environment according to a set of rules to produce the synthetic environmental model, and rendering an image from the synthetic environmental model for display, the image showing the one or more elements from the set of synthetic image elements replacing real-world topography in the environment.

First claim

The invention claimed is:

1. On a computing device, a method of displaying an image, the method comprising:
receiving image data of an environment, the image data comprising depth data and other image information capturing real-world topography of the environment;
detecting a trigger to build a synthetic environmental model based upon the image data of the environment utilizing a three dimensional representation of the environment;
in response to the trigger, obtaining a set of synthetic image elements for use in constructing the synthetic environmental model;
for a voxel space in the three-dimensional representation of the environment to be rendered, fitting one or more elements from the set of synthetic image elements to the three dimensional representation of the environment, the one or more elements being fit to the voxel space according to a set of rules to produce the synthetic environmental model, wherein the voxel space comprises a portion of the image data, and wherein fitting the one or more elements from the set of synthetic image elements comprises replacing the other image information of the real-world topography of the voxel space with the one or more elements from the set of synthetic elements;
after replacing the other image information of the real-world topography of the voxel space with the one or more elements from the set of synthetic image elements, rendering the image from the synthetic environmental model for display by rendering the one or more elements from the set of synthetic elements for the voxel space; and
outputting the image to a display device.

2. The method of claim 1 , wherein the set of synthetic image elements is selected from a plurality of sets of synthetic image elements.

3. The method of claim 2 , wherein the set of synthetic image elements is selected via user selection.

4. The method of claim 2 , wherein the set of synthetic image elements is selected based on an identity of each of one or more real-world objects to be replaced by one or more elements.

5. The method of claim 1 , wherein the synthetic environmental model is utilized to create a game space for a gaming application executed on the computing device.

6. The method of claim 1 , wherein the trigger is generated responsive to one or more of a receiving a request from an application, receiving a user input, and detecting a completion of a scanning operation for determining the three dimensional representation of the environment.

7. The method of claim 1 , wherein fitting an element to the three dimensional representation comprises performing one or more of a rotation and a scaling of the element.

8. The method of claim 1 , further comprising identifying movement of a real-world object and updating the rendered image based on the movement of the real-world object.

9. A computing system, comprising:
a display device;
a camera;
a logic subsystem; and
a storage device configured to hold instructions executable by the logic subsystem to,
receive image data of an environment, the image data comprising depth data and other image information capturing real-world topography of the environment;
detect a trigger to build a synthetic environmental model utilizing a three dimensional representation of an environment;
in response to the trigger, obtain a set of synthetic image elements for use in constructing the synthetic environmental model;
for a voxel space in the three-dimensional representation of the environment to be rendered, fit one or more elements from the set of synthetic image elements to the voxel space according to a set of rules to produce the synthetic environmental model, wherein the voxel space comprises a portion of the image data and wherein fitting the one or more elements from the set of synthetic image elements comprising replacing the other image information of the real-world topography of the voxel space with the one or more elements;
after replacing the other image information of the real-world topography of the voxel space with the one or more elements from the set of synthetic image elements, render an image from the synthetic environmental model for display by rendering the one or more elements from the set of synthetic elements for the voxel space in place of the other image information of the real-world topology of the voxel space; and
output the image to a display device.

10. The computing system of claim 9 , wherein the camera is a first camera, the computing system further comprising one or more additional cameras, at least one of the additional cameras having a different view than the first camera and capturing different image data from the first camera.

11. The computing system of claim 9 , wherein the three dimensional representation of the environment is determined by scanning the environment to acquire image data of the environment from a plurality of perspectives.

12. The computing system of claim 9 , wherein the instructions are further executable to receive user input selecting one or more real-world objects in the environment that are replaced with the one or more elements from the set of synthetic image elements.

13. The computing system of claim 9 , wherein the three dimensional representation of the environment is received from the remote service via a communications subsystem.

14. The computing system of claim 9 , wherein the three dimensional representation of the environment is stored on the storage device.

15. The computing system of claim 9 , wherein the instructions are further executable to identify movement of a real-world object based on data received from the camera and to update the rendered image based on the movement of the real-world object.

16. In a computing device, a method of providing an augmented reality experience, the method comprising:
receiving image data of an environment, the image data comprising depth data and other image information capturing real-world topography of the environment;
detecting a trigger to build a synthetic environmental model utilizing a three dimensional representation of an environment;
in response to the trigger, obtaining a set of synthetic image elements for use in constructing the synthetic environmental model;
for a voxel space in the three-dimensional representation of the environment to be rendered, fitting one or more elements from the set of synthetic image elements to the voxel space according to a set of rules to produce the synthetic environmental model, wherein the voxel space comprises a portion of the image data and wherein the one or more elements from the set of synthetic image elements replace the other image information of the real-world topography of the voxel space;
after replacing the other image information of the real-world topography of the voxel space with the one or more elements from the set of synthetic image elements, rendering an image from the synthetic environmental model for display rendering the one or more elements from the set of synthetic elements for the voxel space in place of the other image information of the real-world topology of the voxel space;
outputting the image to a display device;
identifying movement of a selected real-world object based upon the image data of the environment;
updating the image based on the movement of the selected real-world object by moving corresponding synthetic image elements to align with the selected real world object to form an updated image; and
outputting the updated image to display device.

17. The method of claim 16 , wherein the movement of the real-world object comprises a movement of the s

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Molyneaux Dave
• White Christopher
• Tsao Anson

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2219/2021

  Shape modification · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T15/005

  General purpose rendering architectures · CPC title