Augmented-reality imaging

Patent metadata

Field	Value
Publication number	US-10127725-B2
Application number	US-201514843709-A
Country	US
Kind code	B2
Filing date	Sep 2, 2015
Priority date	Sep 2, 2015
Publication date	Nov 13, 2018
Grant date	Nov 13, 2018

Abstract

A two-dimensional augmentation image is rendered from a three-dimensional model from a first virtual perspective. A transformation is applied to the augmentation image to yield an updated two-dimensional augmentation image that approximates a second virtual perspective of the three-dimensional model without additional rendering from the three-dimensional model.

First claim

The invention claimed is:

1. On an augmented-reality device, a method of providing a mixed-reality experience, the method comprising:
rendering from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a first real-world location;
outputting the two-dimensional augmentation image;
applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective of the virtual camera spatially registered to a second real-world location different than the first real-world location such that the second virtual perspective is different than the first virtual perspective; and
outputting the updated two-dimensional augmentation image.

2. The method of claim 1 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image as a layer of a mixed-reality recording, the mixed-reality recording including a second layer including a visible-light image captured by a point-of-view camera.

3. The method of claim 2 , wherein the transformation is selected so that the updated two-dimensional augmentation image has a virtual perspective simulating a real-world perspective of the point-of-view camera at the second real-world location when the visible-light image is captured.

4. The method of claim 2 , further comprising:
cropping a larger of the updated two-dimensional augmentation image and the visible-light image to match a smaller of the updated two-dimensional augmentation image and the visible light image in the mixed-reality recording.

5. The method of claim 1 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image to a display of the augmented-reality device for visual presentation of the updated two-dimensional augmentation image by the display.

6. The method of claim 5 , wherein the display is an at least partially see-through display, wherein the updated two-dimensional augmentation image is visually presented on the at least partially see-through display while light from a physical space passes through the at least partially see-through display to a user eye.

7. The method of claim 5 , wherein the updated two-dimensional augmentation image has dimensions that are larger than dimensions of a field of view of the display, and the method further comprises, cropping the updated two-dimensional augmentation image.

8. The method of claim 1 , wherein applying a transformation to the augmentation image includes,
for each pixel of the augmentation image, passing a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,
for each virtual position of the three-dimensional transformation model, mapping the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and
for each updated virtual position in the updated three-dimensional transformation model, passing the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.

9. The method of claim 8 , wherein the method further comprises:
if two or more updated virtual positions map to a same pixel, selecting an updated virtual position nearest to the second virtual perspective as the updated virtual position of the same pixel.

10. An augmented-reality device, comprising:
an at least partially see-through display;
a logic machine; and
a storage machine holding instructions executable by the logic machine to:
render from a three-dimensional model a two-dimensional augmentation image, the two-dimensional augmentation image providing a view of the three-dimensional model from a first virtual perspective of a virtual camera spatially registered to a first real-world location;
output the two-dimensional augmentation image;
applying a transformation to the two-dimensional augmentation image to yield an updated two-dimensional augmentation image, the updated two-dimensional augmentation image approximating a different view of the three-dimensional model from a second virtual perspective of the virtual camera spatially registered to a second real-world location different than the first real-world location such that the second virtual perspective is different than the first virtual perspective; and
output the updated two-dimensional augmentation image.

11. The augmented-reality device of claim 10 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image as a layer of a mixed-reality recording, the mixed-reality recording including a second layer including a visible-light image captured by a point-of-view camera.

12. The augmented-reality device of claim 11 , wherein the transformation is selected so that the updated two-dimensional augmentation image has a virtual perspective simulating a real-world perspective of the point-of-view camera at the second real-world location when the visible-light image is captured.

13. The augmented-reality device of claim 11 , wherein the storage machine further holds instructions executable by the logic machine to:
crop a larger of the updated two-dimensional augmentation image and the visible-light image to match a smaller of the updated two-dimensional augmentation image and the visible light image in the mixed-reality recording.

14. The augmented-reality device of claim 10 , wherein outputting the updated two-dimensional augmentation image includes outputting the updated two-dimensional augmentation image to the at least partially see-through display, and wherein the updated two-dimensional augmentation image is visually presented on the at least partially see-through display while light from a physical space passes through the at least partially see-through display to a user eye.

15. The augmented-reality device of claim 14 , wherein the updated two-dimensional augmentation image has dimensions that are larger than dimensions of a field of view of the display, and wherein the storage machine further holds instructions executable by the logic machine to:
crop the updated two-dimensional augmentation image.

16. The augmented-reality device of claim 10 , wherein applying a transformation to the augmentation image includes,
for each pixel of the augmentation image, pass a screen coordinate of the pixel through an inverse view-projection matrix to yield a virtual position in a three-dimensional transformation model from the first virtual perspective,
for each virtual position of the three-dimensional transformation model, map the virtual position to an updated virtual position in an updated three-dimensional transformation model to simulate the second virtual perspective, and
for each updated virtual position in the updated three-dimensional transformation model, pass the updated virtual position of the pixel through a view-projection matrix to yield an updated screen coordinate of the pixel.

17. The augmented-reality device of claim 16 , wherein the storage machine further holds instructions executable by the logic machine to:
if two or more updated virtua

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Kohler Jeffrey
• Demandolx Denis
• Guyman Will
• Michail Ashraf Ayman
• Park Minshik
• Nafziger Justin
• Richards Eric

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/00

  Manipulating three-dimensional [3D] models or images for computer graphics · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title