Cleaning of depth data by elimination of artifacts caused by shadows and parallax

Patent metadata

Field	Value
Publication number	US-10116915-B2
Application number	US-201715407493-A
Country	US
Kind code	B2
Filing date	Jan 17, 2017
Priority date	Jan 17, 2017
Publication date	Oct 30, 2018
Grant date	Oct 30, 2018

Abstract

Multiple Holocam Orbs observe a real-life environment and generate an artificial reality representation of the real-life environment. Depth image data is cleansed of error due to LED shadow by identifying the edge of a foreground object in an (near infrared light) intensity image, identifying an edge in a depth image, and taking the difference between the start of both edges. Depth data error due to parallax is identified noting when associated text data in a given pixel row that is progressing in a given row direction (left-to-right or right-to-left) reverses order. Sound sources are identified by comparing results of a blind audio source localization algorithm, with the spatial 3D model provided by the Holocam Orb. Sound sources that corresponding to identifying 3D objects are associated together. Additionally, types of data supported by a standard movie data container, such as an MPEG container, is expanding to incorporate free viewpoint data (FVD) model data. This is done by inserting FVD data of different individual 3D objects at different sample rates into a single video stream. Each 3D object is separately identified by a separately assigned ID.

First claim

What is claimed is:

1. An environment sensing apparatus, comprising:
a depth image capture device including:
an illumination source, the depth image capture device using reflected illumination from the illumination source to determine a depth map of a scene, the depth map being comprised of lines of depth data points; and
an intensity image capture device that captures an intensity image of the scene, the intensity image being comprised of lines of intensity data points; and

a data processing unit implementing the following steps:
on a line-by-line basis, identifying a first edge of an object in a current line, of intensity data points, of the intensity image, and identifying a corresponding second edge of the same object in a corresponding current line, of depth data points, of the depth map;
defining as an observed shadow, depth data points in the current line of the depth map that lie between a first depth data point, whose line position corresponds to a position of the first edge in the corresponding current line of the intensity image, and a second depth data point that corresponds to a position of the second edge in the same current line of the depth map; and
selectively removing depth data points within the defined observed shadow from the depth map.

2. The environment sensing apparatus of claim 1 , wherein:
the depth image capture device is part of a time-of-flight (TOF) system; and
the illumination source is a near infrared (NIR) light source, and the intensity data points are NIR light intensity data points.

3. The environment sensing apparatus of claim 1 , wherein:
the depth image capture device is part of a structured light system; and
the illumination source projects a known pattern image.

4. The environment sensing apparatus of claim 1 , wherein:
the lines of depth data points are one of rows or columns of depth data points within the depth map; and
the lines of intensity data points are one of rows or columns of intensity data points within the intensity image.

5. The environment sensing apparatus of claim 1 , wherein the step of selectively removing depth data points within the defined observed shadow from the depth map, includes:
calculating a theoretical intensity shadow of the object along the current line of the depth map based on known relative positions of the illumination source and the intensity image capture device; and
removing all depth data points that define the observed shadow in response to the observed shadow not being greater than the theoretical shadow.

6. The environment sensing apparatus of claim 5 , wherein the step of selectively removing depth data points within the defined observed shadow from the depth map, further includes:
not removing any of the depth data points within the defined observed shadow in response to the observed shadow being greater than the theoretical shadow.

7. An environment sensing apparatus, comprising:
a depth image capture device, which captures a depth map of a scene, the depth map being comprised of lines of depth data points; and
an intensity image capture device that captures an intensity image of the scene, the captured intensity image of the scene being a texture image including a plurality of lines of texture data points; and
a data processing unit implementing the following steps:
mapping texture data points from a line of the texture image to corresponding depth data points in a corresponding line of the depth map;
identifying, as a procession direction, a prominent position progression of mapped depth data points to texture data points in succession along a first direction of the line of the texture image;
proceeding one depth data point at a time along the procession direction in the corresponding line of the depth map, in response to a current depth data point being mapped to a texture data point whose position does not follow the prominent position progression, identifying as parallax error depth data all depth data points in the corresponding line of the depth map spanning from the current depth data point to a first depth data point whose mapped texture data does follow the prominent position progression; and
removing from the depth map all depth data points identified as parallax error depth data.

8. The environment sensing apparatus of claim 7 , wherein:
positions of each texture data point along each line of texture data are assigned an incremental numerical identifier increasing along the first direction; and
the first depth data point is the first encountered depth data point along the first direction whose mapped texture data point has a numerical identifier higher than the numerical identifier of the texture data point mapped to the current depth data point.

9. The environment sensing apparatus of claim 7 , wherein:
positions of each texture data point along each line of texture data are assigned an incremental numerical identifier increasing along the first direction;
the prominent position progression is opposite to the first direction along the line of the texture image; and
the first depth data point is the first encountered depth data point in the direction opposite the first direction whose mapped texture data point has a numerical identifier lower than the numerical identifier of the texture data point mapped to the current depth data point.

10. The environment sensing apparatus of claim 7 , wherein:
the depth image capture device is part of a time-of-flight (TOF) system; and
the intensity image capture device is a visual spectrum camera.

11. The environment sensing apparatus of claim 7 , wherein:
the depth image capture device is part of a structured light system; and
the intensity image capture device is a visual spectrum camera.

12. A method of removing erroneous depth data points from a depth map of a scene, comprising:
accessing the depth map of the scene, the depth map being generated by a depth image capture system including an intensity image capture device and using illumination from an illumination source to determine the depth map and an intensity image of the scene, the depth map being comprised of lines of depth data points, and the intensity image being comprised of lines of intensity data points;
on a line-by-line basis, identifying a first edge of an object in a current line, of intensity data points, of the intensity image, and identifying a corresponding second edge of the same object in a corresponding current line, of depth data points, of the depth map;
defining as an observed shadow, depth data points in the current line of the depth map that lie between a first depth data point, whose line position corresponds to a position of the first edge in the corresponding current line of the intensity image, and a second depth data point that corresponds to a position of the second edge in the same current line of the depth map; and
selectively removing depth data points within the defined observed shadow from the depth map.

13. The method of claim 12 , wherein:
the depth image capture system is a time-of-flight (TOF) system; and
the illumination source is a near infrared (NIR) light source, and the intensity data points are NIR light intensity data points.

14. The method of claim 12 , wherein:
the depth image capture system is a structured light system; and
the illumination source projects a known pattern image.

15. The method of claim 12 , wherein:
the lines of depth data points are one of rows or columns of depth data points within the depth map; and
the lines of intensity data points are one of rows or columns of intensity data points within

Assignees

• Seiko Epson Corp

Inventors

• Mogalapalli Vivek
• Moravec Ivo
• Mannion Michael Joseph

Classifications

• G06T7/13

  Edge detection · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/254

  in combination with electromagnetic radiation sources for illuminating objects · CPC title

• H04N13/128

  Adjusting depth or disparity · CPC title

• H04N13/122 Primary

  Improving the three-dimensional [3D] impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues (H04N13/128 takes precedence) · CPC title