Electronic device and operating method of electronic device

Patent metadata

Field	Value
Publication number	US-12430784-B2
Application number	US-202217949439-A
Country	US
Kind code	B2
Filing date	Sep 21, 2022
Priority date	Jan 6, 2022
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Disclosed is an operating method of an electronic device which includes a processor performing machine learning of a monocular depth estimation module. The operating method includes obtaining, by the processor, a first image and a second image respectively photographed by a first camera and a second camera of different locations, inferring, by the processor, a plurality of multi-cyclic disparities by applying weights of the monocular depth estimation module to the first image plural times and calculating a plurality of multi-cyclic loss functions based on the first image, the second image, and the plurality of multi-cyclic disparities, and updating, by the processor, the weights of the monocular depth estimation module through machine learning, based on the plurality of multi-cyclic loss functions.

First claim

What is claimed is:

1. An operating method of an electronic device which includes a processor configured to train a monocular depth estimation module, the method comprising:
obtaining, by the processor, a first image photographed by a first camera and a second image photographed by a second camera;
inferring, by the processor, a plurality of multi-cyclic disparities by applying weights of the monocular depth estimation module to the first image a plurality of times and calculating a plurality of multi-cyclic loss functions based on the first image, the second image, and the plurality of multi-cyclic disparities; and
updating, by the processor, the weights of the monocular depth estimation module through the training based on the plurality of multi-cyclic loss functions,
wherein the calculating of the plurality of multi-cyclic loss functions includes
inferring a (k+1)-th multi-cyclic disparity of the plurality of multi-cyclic disparities from a k-th restored image by applying the weights of the monocular depth estimation module,
generating a (k+1)-th restored image from the second image by using the (k+1)-th multi-cyclic disparity, and
calculating a difference between the first image and the (k+1)-th restored image as at least another of the multi-cyclic loss functions, and

wherein k is a positive integer.

2. The method of claim 1 , wherein the plurality of multi-cyclic disparities are respectively inferred with respect to pixels of the first image.

3. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions further includes, prior to the inferring the (k+1)-th multi-cycle disparity:
inferring a first multi-cyclic disparity of the plurality of multi-cyclic disparities from the first image by applying the weights of the monocular depth estimation module;
generating a first restored image from the second image using the first multi-cyclic disparity; and
calculating a difference between the first image and the first restored image as a first multi-cyclic loss function of the plurality of multi-cyclic loss functions.

4. The method of claim 1 , wherein the updating of the weights of the monocular depth estimation module through the training includes:
updating the weights of the monocular depth estimation module through the training such that differences between the plurality of multi-cyclic disparities decrease.

5. The method of claim 1 , wherein the updating of the weights of the monocular depth estimation module through the training includes:
updating the weights of the monocular depth estimation module through the training such that a sum of the plurality of multi-cyclic disparities decreases.

6. The method of claim 1 , further comprising:
obtaining, by the processor, a third image photographed by a third camera and a fourth image photographed by a fourth camera;
inferring, by the processor, a plurality of second multi-cyclic disparities by applying weights of the monocular depth estimation module to the third image and calculating a plurality of second multi-cyclic loss functions based on the third image, the fourth image, and the plurality of second multi-cyclic disparities; and
updating, by the processor, the weights of the monocular depth estimation module through the training, based on the plurality of second multi-cyclic loss functions.

7. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions includes repeating the inferring of the (k+1)-th multi-cyclic disparity, the generating of the (k+1)-th restored image, and the calculating of the difference between the first image and the (k+1)-th restored image until the “k” reaches a maximum value while sequentially increasing the “k”.

8. The method of claim 7 , wherein the maximum value of the “k” is set by a user.

9. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions includes repeating the inferring of the (k+1)-th multi-cyclic disparity, the generating of the (k+1)-th restored image, and the calculating of the difference between the first image and the (k+1)-th restored image while sequentially increasing the “k” until at least one of the plurality of multi-cyclic loss functions diverges.

10. The method of claim 9 , wherein the weights of the monocular depth estimation module are initialized in response to the at least one of the plurality of multi-cyclic loss functions diverging.

11. The method of claim 1 , wherein the calculating of the plurality of multi-cyclic loss functions further includes:
repeating, at least once, the inferring of the (k+1)-th multi-cyclic disparity, the generating of the (k+1)-th restored image, and the calculating of the difference between the first image and the (k+1)-th restored image; and
calculating differences between the (k+1)-th restored images as others of the multi-cyclic loss functions.

12. An electronic device comprising:
a first camera;
an interface device configured to communicate with an external database; and
a processor configured to execute a monocular depth estimation module, the processor configured to
obtain a first image and a second image from the external database through the interface device,
infer a plurality of multi-cyclic disparities corresponding to pixels of the second image by applying weights of the monocular depth estimation module to the first image and calculate a plurality of multi-cyclic loss functions corresponding to the pixels of the second image based on the first image, the second image, and the plurality of multi-cyclic disparities, and
update the weights of the monocular depth estimation module through machine learning based on the plurality of multi-cyclic loss functions, and

wherein the processor is further configured to
obtain a third image from the first camera,
infer disparities by applying the weights of the monocular depth estimation module to the third image, and
calculate depth data of the third image from the disparities,

wherein, while sequentially increasing “k” (k being a positive integer), the processor is configured to repeat operations
calculating a (k+1)-th multi-cyclic disparity from a k-th restored image by using the weights of the monocular depth estimation module,
generating a (k+1)-th restored image from the first image by using the (k+1)-th multi-cyclic disparity,
calculating a (k+1)-th multi-cyclic loss function based on a difference between the second image and the (k+1)-th restored image and differences between the first to k-th multi-cyclic disparities, and
updating the weights of the monocular depth estimation module through the machine learning based on the (k+1)-th multi-cyclic loss function.

13. The electronic device of claim 12 , further comprising:
a second camera,
wherein the processor is further configured to
obtain a fourth image from the second camera, respectively,
infer a plurality of second multi-cyclic disparities respectively corresponding to pixels of the third image by applying weights of the monocular depth estimation module to the fourth image and calculate a plurality of second multi-cyclic loss functions respectively corresponding to the pixels of the third image based on the third image, the fourth image, and the plurality of second multi-cyclic disparities, and
update the weights of the monocular depth estimation module through the machine learning based on the plurality of second multi-cyclic loss functions.

14. The electronic device of claim 12 , wherein the processor is further configured to, prior to the inferring the (k+1)-th multi-cycle disparity,
calculate a first multi-

Assignees

• Samsung Electronics Co Ltd
• Postech Res & Business Dev Found

Inventors

• Kim Sang Woo
• Kim Bum Jun

Classifications

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20224

  Image subtraction · CPC title

• H04N2013/0081

  Depth or disparity estimation from stereoscopic image signals · CPC title

• H04N13/243

  using three or more two-dimensional [2D] image sensors · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title