Techniques for improving perception of projections of subsurface features on a terrain surface in augmented reality

Patent metadata

Field	Value
Publication number	US-10930078-B1
Application number	US-201816178360-A
Country	US
Kind code	B1
Filing date	Nov 1, 2018
Priority date	Nov 1, 2017
Publication date	Feb 23, 2021
Grant date	Feb 23, 2021

Abstract

In one embodiment, techniques are provided for improving perception of representations of subsurface features (e.g., virtual paint markings) in augmented reality. An input image of a terrain surface is accessed. An augmentation stencil image aligned with the input image is created and represented utilizing HSL color space. The input image is converted to the HSL color space. The technique creates and displays an augmented image that, for each pixel that falls outside of the representation subsurface features, has pixel values based on a hue value, a saturation value and a lightness value of the input image and for each pixel that coincides with the representation subsurface features has pixel values based on a hue value and a saturation value of the augmentation stencil image and a lightness value based on the input image.

First claim

What is claimed is:

1. A method comprising:
accessing an input image of a terrain surface in a physical environment;
creating an augmentation stencil image aligned with the input image based on an orthogonal projection of a model of the subsurface features onto a model of the terrain surface, the augmentation stencil image to include pixels set to one or more virtual paint colors that serve as a representation of projected subsurface features, the augmentation stencil image including pixels that each have pixel values in a hue saturation lightness (HSL) color space;
converting the input image to include pixels that each have pixel values in the HSL color space;
creating, by an electronic device, an augmented image that, for each pixel that falls outside of the representation of projected subsurface features, has pixel values based on a hue value, a saturation value and a lightness value of the input image, and for each pixel that coincides with the representation of projected subsurface features, has pixel values based on a hue value and a saturation value of the augmentation stencil image that provide color and a lightness value based on the input image that preserves texture; and
displaying, on a display device, the augmented image to a user.

2. The method of claim 1 , wherein the electronic device is an augmented reality device, and the method further comprises:
capturing the input image of the terrain surface by a camera of the augmented reality device, and
wherein the display device is a display device of the augmented reality device.

3. The method of claim 2 , wherein the camera that captures the input image has a pose in the physical environment, and the creating the augmentation stencil image comprises:
in a three-dimensional (3D) virtual environment, aligning the model of the subsurface features with the model of the terrain surface;
orthogonally projecting the model of the subsurface feature onto the model of the terrain surface;
positioning a virtual camera in the 3D virtual environment to have a pose in the 3D virtual environment that corresponds to the pose of the camera in the physical environment; and
rendering the augmentation stencil image based on the orthogonal projection of the model of the subsurface features on the model of the terrain surface as viewed from the pose of the virtual camera.

4. The method of claim 1 , wherein the creating the augmented image sets each pixel that falls outside of the representation of projected subsurface features to have pixel values equal to the hue value, the saturation value and the lightness value of the input image.

5. The method of claim 1 , wherein the creating the augmented image sets each pixel that coincides with the representation of projected subsurface features to have pixel values equal to the hue value and the saturation value of the augmentation stencil image.

6. The method of claim 5 , wherein the creating the augmented image sets each pixel that coincides with the representation of projected subsurface features to have pixel values equal to the lightness value of the input image.

7. The method of claim 5 , wherein the creating the augmented image sets each pixel that coincides with the representation of projected subsurface features to have pixel values with a lightness value based on the input image according to a first relation for non-white colors and a second, different relation for white.

8. The method of claim 7 , wherein the first relation is based on a white threshold and a black threshold, and the second relation is based on a lightness threshold.

9. The method of claim 1 , wherein the subsurface features are pipes and the terrain surface is a paved surface.

10. The method of claim 1 , wherein the one or more virtual paint colors is a pattern of multiple virtual paint colors.

11. A method comprising:
accessing an input image of a surface in a physical environment, the input image including pixels that each have pixel values in a color space;
creating an augmentation stencil image aligned with the input image, the augmentation stencil image to include pixels set to one or more virtual paint colors that serve as a representation of hidden features, the augmentation stencil image including pixels that each have pixel values in the color space;
creating, by an electronic device, an augmented image that, for each pixel that falls outside of the representation of hidden features, has pixel values based on the pixel values of the input image, and for each pixel that coincides with the representation of hidden features, has at least one pixel value based on the pixel values of the augmentation stencil image that provides color and at least one pixel value based on the pixel values of the input image that preserves texture; and
displaying, on a display device, the augmented image to a user.

12. The method of claim 11 , wherein the electronic device is an augmented reality device, and the method further comprises:
capturing the input image of the surface by a camera of the augmented reality device, and
wherein the display device is a display device of the augmented reality device.

13. The method of claim 12 , wherein the camera that captures the input image has a pose in the physical environment, and the creating the augmentation stencil image comprises:
in a three-dimensional (3D) virtual environment, aligning a model of the hidden features with a model of the surface;
projecting the model of the hidden features onto the model of the surface;
positioning a virtual camera in the 3D virtual environment to have a pose in the 3D virtual environment that corresponds to the pose of the camera in the physical environment; and
rendering the augmentation stencil image based on the projection of the model of the hidden features on the model of the surface as viewed from the pose of the virtual camera.

14. The method of claim 11 , wherein the color space is a hue saturation lightness (HSL) color space.

15. The method of claim 14 , wherein the creating the augmented image sets each pixel that falls outside of the representation of hidden features to have pixel values based on a hue value, a saturation value and a lightness value of the input image, and for each pixel that coincides with the representation of hidden features to have pixel values based on a hue value and a saturation value of the augmentation stencil image that provide color and a lightness value based on the input image that preserves texture.

16. The method of claim 11 , wherein the color space in a hue saturation value (HSV) color space.

17. The method of claim 11 , wherein hidden features are pipes and the surface is a paved surface.

18. An augmented reality device comprising:
a camera configured to capture an input image of a terrain surface in a physical environment;
a display device configured to display an augmented image to a user;
a processor; and
a memory coupled to the processor and configured to store instructions for an augmented reality application that are executable on the processor, the instructions for the augmented reality application, when executed, operable to:
access an augmentation stencil image aligned with the input image, the augmentation stencil image to include pixels set to one or more virtual paint colors that serve as a representation of subsurface features, and
create the augmented image by setting each pixel that falls outside of the representation of subsurface features to have pixel values equal to the input image, and setting each pixel that coincides with the represent

Assignees

• Bentley Sys Inc

Inventors

• Côté Stéphane
• Marcoux-Ouellet Jade

Classifications

• G06T2219/2012

  Colour editing, changing, or manipulating; Use of colour codes · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title