Occlusion and collision detection for augmented reality applications

Patent metadata

Field	Value
Publication number	US-2022230383-A1
Application number	US-202217714918-A
Country	US
Kind code	A1
Filing date	Apr 6, 2022
Priority date	Oct 7, 2019
Publication date	Jul 21, 2022
Grant date	—

Abstract

Techniques for occlusion and collision detection in an AR session are described. In an example, a depth sensor is used to generate a depth image. Distortions in the depth image are reduced or eliminated by at least dividing the depth image into depth layers and moving depth pixels between the layers. An RGBD image is generated from the depth image, as updated, and an RGB image generated at substantially the same time as the depth image. Occlusion of a virtual object is detected based on the RGBD image. Further, a 3D model of the real-world environment is generated from the depth images, as updated, and includes multi-level voxels. Collision with the virtual object is detected based on the multi-level voxels. Rendering of the virtual object in an AR session is based on the occlusion and collision detection.

First claim

What is claimed is:

1 . A method implemented by a computer system, the method including:
generating, in an augmented reality (AR) session and based on a depth sensor of the computer system, a depth image; dividing the depth image into depth layers, each depth layer corresponding to a depth range and including pixels having depth values within the depth range; selecting, from the depth layers, a first depth layer having a first layer number and a second depth layer having a second layer number; adjusting the first depth layer based on the first layer number, first pixels in the first depth layer, the second layer number, and second pixels in the second depth layer, wherein the adjusting includes moving a pixel from the second depth layer to the first depth layer; updating the depth image based on the adjusting; and outputting the depth image as updated to at least one AR application associated with the AR session.

2 . The method of claim 1 , wherein a total number of the depth layers is based on a maximum depth of the depth sensor.

3 . The method of claim 1 , wherein a difference between depth ranges of two consecutive depth layers is between 0.4 meters and 0.6 meters.

4 . The method of claim 1 , wherein the first depth layer and the second depth layer are selected based on a difference between the first layer number and the second layer number being equal to or larger than two.

5 . The method of claim 4 , wherein the first depth layer and the second depth layer are selected further based on each of a total number of the first pixels and a total number of the second pixels being equal to or larger than a predefined threshold number.

6 . The method of claim 1 , wherein the first layer number is larger than the second layer number, and wherein adjusting the first depth layer includes performing a morphological dilation from the first depth layer to the second depth layer.

7 . The method of claim 6 , wherein a size of a kernel of the morphological dilation is based on a difference between the first layer number and the second layer number.

8 . The method of claim 6 , wherein the morphological dilation is iteratively repeated for a number of iterations, and wherein the number of iterations is based on a difference between the first layer number and the second layer number.

9 . The method of claim 1 , further including:
generating, in the AR session and based on a red, green, and blue (RGB) optical sensor of the computer system, an RGB image; generating, based on the depth image as updated and the RGB image, an RGB depth (RGBD) image; generating, based on the depth image as updated, a set of three dimensional (3D) points in a coordinate system of the AR session; generating, based on the depth image, a 3D model that includes multi-level voxels, wherein a multi-level voxel of the multi-level voxels is associated with a 3D point from the set; determining a collision between a virtual object and the multi-level voxel; and rendering, in the AR session, the virtual object based on a depth of the virtual object and the RGBD image and based on the collision.

10 . A computer system including:
a depth sensor configured to generate a depth image in an augmented reality (AR) session; a red, green, and blue (RGB) optical sensor configured to generate an RGB image in the AR session; one or more processors; and one or more memories storing computer-readable instructions that, upon execution by the one or more processors, configure the computer system to: update the depth image by at least dividing the depth image into depth layers and moving a pixel from a first depth layer to a second depth layer of the depth layers; generate, based on the depth image as updated and the RGB image, an RGB depth (RGBD) image; generate, based on the depth image as updated, a set of three dimensional (3D) points in a coordinate system of the AR session; generate a 3D model that includes multi-level voxels, wherein a multi-level voxel of the multi-level voxels is associated with a 3D point from the set; determine a collision between a virtual object and the multi-level voxel; and render, in the AR session, the virtual object based on a depth of the virtual object and the RGBD image and based on the collision.

11 . The computer system of claim 10 , wherein each depth layer corresponds to a depth range and includes pixels having depth values within the depth range, and wherein updating the depth image further includes:
selecting, from the depth layers, the first depth layer and the second depth layer based on a first layer number of the first depth layer and on a second layer number of the second depth layer; and adjusting the second depth layer based on the first layer number, first pixels in the first depth layer, the second layer number, and second pixels in the second depth layer, wherein the adjusting includes moving the pixel from the first depth layer to the second depth layer.

12 . The computer system of claim 10 , wherein generating the RGBD image includes:
registering the depth image with the RGB image based on an image resolution of the depth image, an image resolution of the RGB image, and a transformation between the depth sensor and the RGB optical sensor; performing a depth densification on the depth image, the depth densification including a plurality of morphological dilation on the depth image; filtering, subsequent to the depth densification, the depth image based on a median filter; and up-sampling the depth image as filtered to the image resolution of the RGB image based on the registering, wherein a pixel in the RGBD image corresponds to pixel in the RGB image and a pixel in the depth image as up-sampled.

13 . The computer system of claim 10 , wherein generating the RGBD image includes:
generating an alpha map from the depth image; and up-sampling the depth image and the alpha map to an image resolution of the RGB image.

14 . The computer system of claim 13 , wherein rendering the virtual object includes:
determining that a pixel to be rendered in an AR image corresponds to a first pixel of the RGBD image and to a second pixel of the virtual object; determining, from the RGBD image, a first depth of the first pixel; determining that the first depth is smaller than or equal to a second depth of the second pixel; generating a smoothing factor for the first pixel based on the alpha map; and setting an RGB value for the pixel in the AR image based on a first RGB value of the first pixel, a second RGB value of the second pixel, and the smoothing factor.

15 . The computer system of claim 14 , wherein the smoothing factor is set as α=1−m i /255, and wherein the RGB value is set as c i r =(1−α)c i +αc i o , and wherein “α” is the smoothing factor, “i” is the pixel, a “m i ” is a value determined for the pixel from the alpha map, “c i r ” is the RGB value, “c i ” is the first RGB value, and “c i o ” is the second RGB value.

16 . The computer system of claim 10 , wherein rendering the virtual object includes:
determining that a pixel to be rendered in an AR image corresponds to a first pixel of the RGBD image and to a second pixel of the virtual object; determining, from the RGBD image, a first depth of the first pixel; determining that the first depth is larger than a second depth of the second pixel; and setting an RGB value for the pixel in the AR image to be equal to an RGB value of the second pixel.

17 . One or more non-transitory computer-storage media storing instructions that, upon execution on a computer

Assignees

• Guangdong Oppo Mobile Telecommunications Corp Ltd

Inventors

• Tian Yuan
• Xu Yi
• Ma Yuxin
• Quan Shuxue

Classifications

• G06T15/40 Primary

  Hidden part removal · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title