Systems and methods for generating composite depth images based on signals from an inertial sensor

Patent metadata

Field	Value
Publication number	US-11082607-B2
Application number	US-202016899398-A
Country	US
Kind code	B2
Filing date	Jun 11, 2020
Priority date	Jan 22, 2019
Publication date	Aug 3, 2021
Grant date	Aug 3, 2021

Abstract

Systems and methods for generating composite depth images are disclosed. Exemplary implementations may: capture, by a depth sensor, a set of depth images over a capture period of time; generate, by an inertial sensor, inertial signals that convey values of one or more inertial parameters characterizing motion of the depth sensor during the capture period of time; select a target capture position based on one or more of the capture positions of the set of depth images; generate, using the values of the one or more inertial parameters during the capture period of time, re-projected depth images; and generate a composite depth image by combining multiple depth images, such multiple depth images including a first re-projected depth image and a second re-projected depth image.

First claim

What is claimed is:

1. A system configured to generate composite depth images, the system comprising:
a depth sensor configured to capture a set of depth images over a capture period of time, the set of depth images including depth information, the depth information of the individual depth images indicating distance from capture positions of the set of depth images to surfaces viewable from the capture positions, wherein the set of depth images includes at least a first depth image captured from a first capture position, and a second depth image captured from a second capture position that is different from the first capture position;
an inertial sensor configured to generate inertial signals that convey values of one or more inertial parameters characterizing motion of the depth sensor during the capture period of time; and
one or more hardware processors configured by machine-readable instructions to:
select a target capture position based on one or more of the capture positions of the set of depth images;
generate, using the values of the one or more inertial parameters, a first re-projected depth image representing the depth information included in the first depth image as if the first depth image had been captured from the target capture position;
generate, using the values of the one or more inertial parameters, a second re-projected depth image representing the depth information included in the second depth image as if the second depth image had been captured from the target capture position; and
generate a composite depth image by combining multiple depth images, such multiple depth images including one or both of the first re-projected depth image and the second re-projected depth image.

2. The system of claim 1 , wherein the set of depth images further includes a third depth image captured at a third time from a third capture position different from the first capture position and the second capture position, wherein the target capture position is the third capture position, and wherein generating the composite depth image is further based on the third depth image.

3. The system of claim 2 , wherein generating the composite depth image by combining multiple depth images is based on median values corresponding to a particular three-dimensional point in space, wherein the median values are based on multiple depth images.

4. The method of claim 3 , further comprising:
presenting images on a display of an augmented reality device to a user such that the user can view reality and the images simultaneously, wherein the images are based at least in part on information included in the composite depth image.

5. The system of claim 1 , further comprising a sensor attached to the depth sensor, wherein the sensor has a particular position, wherein the target capture position is the particular position of the sensor.

6. The system of claim 1 , wherein the one or more hardware processors are further configured by machine-readable instructions to:
determine the one or more inertial parameters based on the inertial signals, wherein the one or more inertial parameters include a first set of inertial parameters that correspond to the first depth image, and a second set of inertial parameters that correspond to the second depth image;
determine a first rotational change between a rotation of the depth sensor at the target capture position and a first rotation of the depth sensor at the first capture position based on the first set of inertial values; and
determine a second rotational change between the rotation of the depth sensor at the target capture position and a second rotation of the depth sensor at the second capture position based on the second set of inertial values;
wherein generation of the first re-projected depth image is based on the first rotational change, and wherein generation of the second re-projected depth image is based on the second rotational change.

7. The system of claim 6 , wherein the one or more hardware processors are further configured by machine-readable instructions to:
determine a first positional change between the target capture position and the first capture position based on the first set of inertial parameters;
determine a second positional change between the target capture position and the second capture position based on the second set of inertial parameters;
wherein generation of the first re-projected depth image is further based on the first positional change, and wherein generation of the second re-projected depth image is further based on the second positional change.

8. The system of claim 1 , wherein the inertial sensor includes either a gyroscope or an accelerometer, and wherein the one or more inertial parameters include either angular velocity or acceleration, respectively.

9. The system of claim 1 , wherein an individual value from the one or more inertial parameters corresponds to the target capture position.

10. The system of claim 1 , wherein generating the composite depth image includes aggregating depth information corresponding to a particular three-dimensional point in space from the first depth image with depth information from the second depth image that corresponds to the same particular three-dimensional point in space.

11. The system of claim 10 , wherein aggregating the depth information is based on weighted averages of values from multiple depth images.

12. The system of claim 1 , wherein generating the composite depth image includes filtering out the depth information from an individual depth image that is not supported by the depth information from other depth images.

13. The system of claim 1 , further comprising an augmented reality device including a display, wherein the one or more hardware processors are further configured by machinereadable instructions to:
present images on the display to a user such that the user can view reality and the images simultaneously, wherein the images are based at least in part on information included in the composite depth image.

14. A computer-implemented method for generating composite depth images, the method being implemented in a computer system that includes one or more hardware processors configured by machine-readable instructions, the method comprising:
capturing, by a depth sensor, a set of depth images over a capture period of time, the set of depth images including depth information, the depth information of the individual depth images indicating distance from capture positions of the set of depth images to surfaces viewable from the capture positions, wherein the set of depth images includes at least a first depth image captured from a first capture position, and a second depth image captured from a second capture position that is different from the first capture position;
generating, by an inertial sensor, inertial signals that convey values of one or more inertial parameters characterizing motion of the depth sensor during the capture period of time;
selecting a target capture position based on one or more of the capture positions of the set of depth images;
generating, using the values of the one or more inertial parameters, re-projected depth images, wherein the re-projected depth images include a first re-projected depth image and a second re-projected depth image, wherein the first re-projected depth image represents the depth information included in the first depth image as if the first depth image had been captured from the target capture position, and wherein the second re-projected depth image represents the depth information included in the second depth image as if the second depth image had been captured fr

Assignees

• Rpx Corp
• Facebook Tech Llc

Inventors

• Ricci Luca
• Wagner Daniel
• Hol Jeroen Diederik
• Zillner Jakob

Classifications

• G01B11/24 Primary

  for measuring contours or curvatures · CPC title

• H04N23/676 Primary

  Bracketing for image capture at varying focusing conditions · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• G01C21/166

  Mechanical, construction or arrangement details of inertial navigation systems · CPC title

• H04N13/271 Primary

  wherein the generated image signals comprise depth maps or disparity maps · CPC title