Systems and methods for generating depth information from low-resolution images

Patent metadata

Field	Value
Publication number	US-12413859-B2
Application number	US-202318502980-A
Country	US
Kind code	B2
Filing date	Nov 6, 2023
Priority date	Apr 14, 2021
Publication date	Sep 9, 2025
Grant date	Sep 9, 2025

Abstract

A system for generating depth information from low-resolution images is configured to access a plurality of image frames capturing an environment, identify a first group of image frames from the plurality of image frames, and generate a first image comprising a first composite image of the environment using the first group of image frames as input. The first composite image has an image resolution that is higher than an image resolution of the image frames of the first group of image frames. The system is also configured to obtain a second image of the environment, where parallax exists between a capture perspective associated with the first image and a capture perspective associated with the second image. The system is also configured to generate depth information for the environment based on the first image and the second image.

First claim

We claim:

1. A system for generating depth information from low-resolution images of a captured environment, the system comprising:
one or more processors; and
one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the system to generate depth information from low-resolution images of a captured environment by configuring the system to:
access a plurality of image frames capturing an environment and acquired by one or more image capture devices;
identify a first group of image frames from the plurality of image frames;
generate a first image comprising a first composite image of the environment using the first group of image frames as input and using super-resolution imaging techniques, the first composite image comprising an image resolution that is higher than an image resolution of the image frames of the first group of image frames;
obtain a second image of the environment, wherein parallax exists between a capture perspective associated with the first composite image and a capture perspective associated with the second image, wherein the second image comprises an image resolution that is higher than the image resolution of the image frames of the first group of image frames; and
generate depth information for the environment based on the first composite image and the second image.

2. The system of claim 1 , wherein the system comprises a vehicle, and wherein the vehicle comprises an image capture device that captures the plurality of image frames while traveling within the environment.

3. The system of claim 1 , wherein the image frames of the first group of image frames are consecutively captured image frames.

4. The system of claim 1 , wherein the image frames of the first group of image frames are identified based on pose associated with the image frames of the first group of image frames.

5. The system of claim 1 , wherein the second image comprises a second composite image of the environment, the second composite image being generated based on a second group of image frames capturing the environment, the second composite image comprising an image resolution that is higher than an image resolution of the image frames of the second group of image frames.

6. The system of claim 5 , wherein the second group of image frames is captured by a second image capture device that is untethered from a first image capture device that captures the first group of image frames.

7. The system of claim 5 , wherein the second group of image frames is captured by a same image capture device that captures the first group of image frames.

8. The system of claim 7 , wherein the second group of image frames is captured temporally subsequent to the first group of image frames.

9. The system of claim 1 , wherein generating the first image includes using pose values associated with the first group of image frames as input.

10. The system of claim 1 , wherein generating the first image includes using an additional image frame as input, wherein the additional image frame is captured by a second image capture device that is untethered from a first image capture device that captures the first group of image frames.

11. The system of claim 1 , wherein generating the first image includes using input based on coarse depth information determined from the first group of image frames.

12. The system of claim 1 , wherein generating the first image includes rolling shutter correction processing.

13. The system of claim 1 , wherein generating the first image includes motion blur compensation processing.

14. The system of claim 1 , wherein the instructions are executable by the one or more processors to further configure the system to:
obtain second depth information for the environment, the depth information and the second depth information representing an overlapping portion of the environment; and
generate a composite mesh of the environment based on the depth information and the second depth information.

15. The system of claim 14 , wherein the second depth information is obtained based on groups of image frames captured by a same image capture device that captures the first group of image frames.

16. An image capture device, comprising:
an image sensor;
one or more processors; and
one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the image capture device to capture images of an environment to facilitate generation of depth information for the environment by configuring the image capture device to:
capture a plurality of image frames of an environment using the image sensor;
select a first group of image frames from the plurality of image frames;
generate a first composite image of the environment using the first group of image frames as input and using super-resolution imaging techniques, the first composite image comprising an image resolution that is higher than an image resolution of the image frames of the first group of image frames, wherein generating the first composite image includes using an additional image frame as input, wherein the additional image frame is captured by a second image capture device that is untethered from the image capture device; and
(i) transmit the first composite image to a remote device while refraining from transmitting the first group of image frames to the remote device, or (ii) store the first composite image in a nonvolatile manner while refraining from storing the first group of image frames in a nonvolatile manner.

17. The image capture device of claim 16 , wherein the image capture device is connected to a vehicle traveling within the environment during capture of the plurality of image frames.

18. The image capture device of claim 16 , wherein the image frames of the first group of image frames are consecutively captured image frames.

19. The image capture device of claim 16 , wherein the image frames of the first group of image frames are identified based on pose associated with the image frames of the first group of image frames.

20. The image capture device of claim 16 , wherein generating the first composite image includes using pose values associated with the first group of image frames as input.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Bleyer Michael
• Edmonds Christopher Douglas
• Matakos Antonios
• Price Raymond Kirk

Classifications

• H04N23/689

  Motion occurring during a rolling shutter mode · CPC title

• G06T3/4076

  using the original low-resolution images to iteratively correct the high-resolution images · CPC title

• G06T7/55

  from multiple images · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• H04N23/682 Primary

  Vibration or motion blur correction · CPC title