Method and system for unsupervised prediction of image depth and confidence map

Patent metadata

Field	Value
Publication number	US-12100173-B2
Application number	US-202117451663-A
Country	US
Kind code	B2
Filing date	Oct 21, 2021
Priority date	Oct 24, 2020
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

Depth estimation of images using deep learning methods is a wide range of application in Augmented Reality, 3D graphics and robotics. Conventional methods are supervised, which requires explicit ground truth depth information for training and the conventional unsupervised methods fails to provide a generalized solution. The present disclosure estimates accurate depth information and confidence map of a given monocular image in an unsupervised manner. A depth Neural Network receives a monocular image and predicts per-pixel depth map and a confidence map. The depth NN utilizes a negative exponential of photometric loss as ground truth information. The predicted confidence-map is further used to estimate per-pixel uncertainty map. The pose NN predicts a plurality of pose vectors between a plurality of the consecutive monocular images. Finally, the Bayesian inference module is computes the fused depth information and the fused uncertainty map.

First claim

What is claimed is:

1. A processor implemented method, the method comprising:
receiving, by one or more hardware processors, a plurality of monocular images from an imaging device, wherein the plurality of monocular images comprises consecutive image frames;
predicting, by a trained depth neural network executed by via the one or more hardware processors, a depth information and a confidence map based on a monocular image from the plurality of monocular images, wherein training the depth neural network comprises:
receiving a rectified stereo image pair comprising a left image and a right image;
computing a plurality of disparity maps based on the rectified stereo image pair, wherein the plurality of disparity maps comprises a right to left disparity and a left to right disparity;
computing the depth information based on a plurality of parameters comprising a baseline distance of the rectified stereo image, a rectified camera focal length and the plurality of disparity maps;
reconstructing the received rectified stereo image pair based on the depth information, wherein the left image is reconstructed using the right image and the right to left disparity and wherein the right image is reconstructed using the left image and the left to right disparity;
computing a photometric loss by comparing the reconstructed stereo image pair and the received rectified stereo image pair;
computing a negative exponential of the photometric loss, wherein the negative exponential provides a ground truth confidence information;
computing the confidence map based on the negative exponential, wherein a pixel with large photometric error maps to a smallest confidence value;
simultaneously computing natural log of the confidence map to obtain an uncertainty map associated with the confidence map; and
training the depth neural network based on the computed confidence map, the depth information, and the plurality of disparity maps;

predicting, by a trained pose neural network executed by via the one or more hardware processors, a pose information associated with the plurality of monocular image sequences based on the depth information learnt by the trained pose neural network, wherein training the pose neural network comprises:
receiving a plurality of consecutive monocular images from the imaging device;
computing a plurality of pose vectors based on the plurality of consecutive monocular images;
reconstructing the plurality of consecutive monocular images based on the depth information and the plurality of pose vectors, wherein the corresponding depth information is computed using the depth neural network;
computing a temporal loss information by comparing the plurality of reconstructed consecutive monocular images and the plurality of received consecutive monocular images; and
training the pose neural network based on the computed loss information; and

computing, by the one or more hardware processors, a fused data by combining the predicted data and a propagated data, wherein the fused data comprises a combined depth map and a combined confidence map, wherein the predicted data comprising the predicted depth information, the uncertainty map and the pose information associated with the monocular image, by using Bayesian Inference model by:
receiving the propagated data associated with a previous frame, wherein the propagated data comprises a propagated depth information and a propagated uncertainty map associated with the previous frame; and
combining the propagated data with the predicted data to obtain the fused data, wherein the predicted data of the current image frame is propagated to the next image frame.

2. The method of claim 1 ,
wherein the plurality of pose vectors comprises translation and rotation information of the imaging device.

3. A system comprising:
at least one memory storing programmed instructions; one or more Input/Output (I/O) interfaces; and one or more hardware processors operatively coupled to the at least one memory, wherein the one or more hardware processors are configured by the programmed instructions to:
receive a plurality of monocular images from an imaging device, wherein the plurality of monocular images comprises consecutive image frames;
predict, by a trained depth neural network, a depth information and a confidence map based on a monocular image from the plurality of monocular images, wherein training the depth neural network comprises:
receiving a rectified stereo image pair comprising a left image and a right image;
computing a plurality of disparity maps based on the rectified stereo image pair, wherein the plurality of disparity maps comprises a right to left disparity and a left to right disparity;
computing the depth information based on a plurality of parameters comprising a baseline distance of the rectified stereo image, a rectified camera focal length and the plurality of disparity maps;
reconstructing the received rectified stereo image pair based on the depth information, wherein the left image is reconstructed using the right image and the right to left disparity and wherein the right image is reconstructed using the left image and the left to right disparity;
computing a photometric loss by comparing the reconstructed stereo image pair and the received rectified stereo image pair;
computing a negative exponential of the photometric loss, wherein the negative exponential provides a ground truth confidence information;
computing the confidence map based on the negative exponential;
simultaneously computing natural log of the confidence map to obtain an uncertainty map associated with the confidence map; and
training the depth neural network based on the computed confidence map, the depth information and the plurality of disparity maps;

predict a pose information associated with the plurality of monocular image sequences based on the depth information by a trained pose neural network, wherein training the pose neural network comprises:
receiving a plurality of consecutive monocular images from the imaging device;
computing a plurality of pose vectors based on the plurality of consecutive monocular images;
reconstructing the plurality of consecutive monocular images based on the depth information and the plurality of pose vectors, wherein the corresponding depth information is computed using the depth neural network;
computing a temporal loss information by comparing the plurality of reconstructed consecutive monocular images and the plurality of received consecutive monocular images; and
training the pose neural network based on the computed loss information; and

compute a fused data by combining the predicted data and a propagated data, wherein the fused data comprises a combined depth map and a combined confidence map, wherein the predicted data comprising the predicted depth information, the uncertainty map and the pose information associated with the monocular image, by using Bayesian Inference model by:
receiving the propagated data associated with a previous frame, wherein the propagated data comprises a propagated depth information and a propagated uncertainty map associated with the previous frame; and
combining the propagated data with the predicted data to obtain the fused data, wherein the predicted data of the current image frame is propagated to the next image frame.

4. The system of claim 3 ,
wherein the plurality of pose vectors comprises translation and rotation information of the imaging device.

5. One or more non-transitory machine readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors causes:
receiving, by one or more hardware processors, a plurality of monocular images from an imaging device, wherein the plurality of monocular images compri

Assignees

• Tata Consultancy Services Ltd
• Tata Consultancyservices Ltd

Inventors

• Bhutani Vishal Kumar
• Vankadari Madhu Babu
• Dutta Anima Majumder
• Jha Omprakash Manojkumar
• Dutta Samrat

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title