Planar surface detection apparatus and method

Patent metadata

Field	Value
Publication number	US-12367599-B2
Application number	US-202217698577-A
Country	US
Kind code	B2
Filing date	Mar 18, 2022
Priority date	Oct 27, 2021
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

Provided is a method and apparatus for detecting a planar surface, the method including acquiring, based on a pixelwise disparity of an input image estimated in a first network, a pixelwise plane parameter of the input image, determining a pixelwise segment matching probability of the input image based on a second network trained to perform a segmentation of an image, acquiring a segment-wise plane parameter based on the pixelwise plane parameter and the pixelwise segment matching probability, and detecting a planar surface in the input image based on the segment-wise plane parameter.

First claim

What is claimed is:

1. A method of detecting a planar surface, the method comprising:
acquiring, based on a pixelwise disparity of an input image estimated in a first network, a pixelwise plane parameter of the input image;
determining a pixelwise segment matching probability of the input image based on a second network trained to perform a segmentation of an image;
acquiring a segment-wise plane parameter by calculating a weighted sum of pixelwise plane parameter based on the pixelwise segment matching probability corresponding to each segment in the image; and
detecting a planar surface in the input image based on the segment-wise plane parameter.

2. The method of claim 1 , wherein the detecting of the planar surface in the input image comprises:
acquiring pixelwise segment clustering information based on the pixelwise segment matching probability; and
detecting the planar surface in the input image based on the segment-wise plane parameter and the pixelwise segment clustering information.

3. The method of claim 1 , wherein the acquiring of the segment-wise plane parameter for each segment in the input image comprises:
obtaining, based on the second network, the weighted sum of the pixelwise plane parameter based on the pixelwise segment matching probability corresponding to the corresponding segment; and
acquiring a plane parameter of the corresponding segment based on the weighted sum of the pixelwise plane parameter.

4. The method of claim 1 , wherein the first network and the second network are trained based on at least one of:
a first loss associated with a probability that each pixel matches each segment, which is calculated based on a probability distribution of a plane parameter corresponding to each segment clustered based on the second network; or
a second loss associated with a difference between a first image and an image obtained by converting a second image corresponding to the first image based on a disparity estimated in the first network to correspond to the first image.

5. The method of claim 1 , wherein the acquiring of the pixelwise plane parameter comprises:
determining the pixelwise disparity of the input image based on the first network; and
acquiring the pixelwise plane parameter comprising a normal vector and distance information from the pixelwise disparity based on an intrinsic parameter of a camera that captures the input image.

6. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

7. A learning method of a plane detection model, the learning method comprising:
acquiring a pixelwise plane parameter of a first image included in training data based on a pixelwise disparity of the first image acquired by applying the first image to a first network;
determining a pixelwise segment matching probability of the first image by applying the first image to a second network; and
training the first network and the second network based on a first loss associated with a probability that each pixel of the first image matches each segment, which is calculated from a weighted sum of the pixelwise plane parameter based on pixelwise segment matching probability of the first image.

8. The learning method of claim 7 , wherein the probability that each pixel of the first image corresponds to each segment is calculated based on the pixelwise plane parameter and a probability distribution of plane parameters corresponding to a number of segments.

9. The learning method of claim 8 , wherein the probability distribution of the plane parameters corresponding to the number of segments comprises:
a representative value of the plane parameters corresponding to the segments calculated based on the pixelwise segment matching probability and the pixelwise plane parameter; and
a variance of the plane parameters corresponding to the segments calculated based on the pixelwise segment matching probability, the pixelwise plane parameter, and the representative value of the plane parameters corresponding to the segments.

10. The learning method of claim 7 , wherein the training of the first network and the second network comprises:
converting a second image captured at a different viewpoint from that of the first image based on a depth estimated to correspond to the first image in the first network; and
training the first network and the second network based on the first loss and a second loss associated with a difference between the first image and an image obtained through the converting of the second image.

11. The learning method of claim 7 , wherein the training data comprises at least one of:
the first image corresponding to a first monocular image of a stereo image and a second image corresponding to a second monocular image of the stereo image; or
the first image corresponding to a first frame of a video image and a second image corresponding to a second frame of the video image.

12. The learning method of claim 7 , wherein the acquiring of the pixelwise plane parameter comprises:
estimating the pixelwise disparity of the first image by applying the first image to the first network; and
acquiring the pixelwise plane parameter comprising a normal vector and distance information from the pixelwise disparity based on an intrinsic parameter of a camera that captures the first image.

13. An apparatus for detecting a planar surface, the apparatus comprising:
a processor configured to:
acquire, based on a pixelwise disparity of an input image estimated in a first network, a pixelwise plane parameter of the input image;
determine a pixelwise segment matching probability of the input image based on a second network trained to perform a segmentation of an image;
acquire a segment-wise plane parameter by calculating a weighted sum of the pixelwise plane parameter based on the pixelwise segment matching probability corresponding to each segment in the image; and
detect a planar surface in the input image based on the segment-wise plane parameter.

14. The apparatus of claim 13 , wherein the processor is further configured to:
acquire pixelwise segment clustering information based on the pixelwise segment matching probability; and
detect the planar surface in the input image based on the segment-wise plane parameter and the pixelwise segment clustering information.

15. The apparatus of claim 13 , wherein the processor is further configured to:
obtain, based on the second network, the weighted sum of the pixelwise plane parameter based on the pixelwise segment matching probability corresponding to the corresponding segment; and
acquire a plane parameter of the corresponding segment based on the weighted sum of the pixelwise plane parameter.

16. The apparatus of claim 13 , wherein the first network and the second network are trained based on at least one of:
a first loss associated with a probability that each pixel matches each segment, which is calculated based on a probability distribution of a plane parameter corresponding to each segment clustered based on the second network; or
a second loss associated with a difference between a first image and an image obtained by converting a second image corresponding to the first image based on a disparity estimated in the first network to correspond to the first image.

17. The apparatus of claim 13 , wherein the processor is further configured to:
determine the pixelwise disparity of the input image based on the first network; and
acquire the pixelwise plane parameter comprising a norm

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Seungeon
• Chang Hyun Sung
• Lee Wonhee
• Jang Seokhwan

Classifications

• G06T2207/20228

  Disparity calculation for image-based rendering · CPC title

• G06T2207/20224

  Image subtraction · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06N3/08

  Learning methods · CPC title