Apparatus and method for providing 3-dimensional around view

Patent metadata

Field	Value
Publication number	US-2020007844-A1
Application number	US-201916559170-A
Country	US
Kind code	A1
Filing date	Sep 3, 2019
Priority date	Jul 11, 2019
Publication date	Jan 2, 2020
Grant date	—

Abstract

A three-dimensional around view providing apparatus for providing a 3D around view through a user interface module included in a vehicle may include a plurality of image pickup units mounted in the vehicle, a depth estimator configured to receive a plurality of images from the plurality of image pickup units and to acquire a plurality of depth maps corresponding to the plurality of images, a controller configured to minimize a depth difference between a first boundary region of a first depth map and a second boundary region of a second depth map. At least one among an autonomous vehicle, a user terminal, and a server according to embodiments of the present disclosure may be associated or integrated with an artificial intelligence (AI) module, a drone (unmanned aerial vehicle (UAV)), a robot, an augmented reality (AR) device, a virtual reality (VR) device, a 5th-generation (5G) service related device, and the like.

First claim

What is claimed is:

1 . A three-dimensional (3D) around view providing apparatus for providing a 3D around view through a user interface module included in a vehicle, the 3D around view providing apparatus comprising:
a plurality of image pickup units mounted in the vehicle; a depth estimator configured to receive a plurality of images from the plurality of image pickup units and to acquire a plurality of depth maps corresponding to the plurality of images; an image processor configured to minimize a depth difference between a first boundary region of a first depth map and a second boundary region of a second depth map, to acquire a first around view depth map with the minimized depth difference between the first boundary region and the second boundary region, and to restore a 3D around view image for the plurality of images by using the first around view depth map, the first boundary region and the second boundary region being boundary regions in which the first depth map and the second depth map, which are adjacent to each other among the plurality of depth maps, overlap; and a display unit configured to display the 3D around view image.

2 . The 3D around view providing apparatus according to claim 1 , wherein
the depth estimator acquires the plurality of depth maps corresponding to the plurality of images by inputting the plurality of images to a deep neural network (DNN) encoder.

3 . The 3D around view providing apparatus according to claim 1 , wherein
the image processor determines the first boundary region and the second boundary region in which the first depth map and the second depth map overlap, based on locations in which the plurality of image pickup units is mounted.

4 . The 3D around view providing apparatus according to claim 1 , wherein
the image processor minimizes the depth difference between the first boundary region and the second boundary region by using a function that minimizes a Euclidean distance between first boundary region 3D coordinate values calculated by a depth value in the first boundary region and second boundary region 3D coordinate values calculated by a depth value in the second boundary region.

5 . The 3D around view providing apparatus according to claim 4 , wherein
the first boundary region 3D coordinate values are 3D coordinate values set to correspond to a focal length and a principal point of each of the image pickup units, based on the depth value in the first boundary region and first boundary region two-dimensional (2D) coordinate values, and the second boundary region 3D coordinate values are 3D coordinate values set to correspond to a focal length and a principal point of each of the image pickup units, based on the depth value in the second boundary region and second boundary region 2D coordinate values.

6 . The 3D around view providing apparatus according to claim 1 , wherein
the image processor minimizes a depth difference between a third boundary region of a third depth map and a fourth boundary region of the first depth map, acquires a second around view depth map with the minimized depth difference between the first boundary region and the second boundary region and the minimized depth difference between the third boundary region and the fourth boundary region, and restores a 3D around view image for the plurality of images by using the second around view depth map, wherein the third boundary region and the fourth boundary region are boundary regions in which the third depth map and the first depth map among the plurality of depth maps overlap, and one side of the first depth map overlaps the second depth map, and the other side of the first depth map overlaps the third depth map.

7 . The 3D around view providing apparatus according to claim 6 , wherein
the image processor calculates a first linear function that minimizes a Euclidean distance between first boundary region 3D coordinate values calculated by a depth value in the first boundary region and second boundary region 3D coordinate values calculated by a depth value in the second boundary region, and a second linear function that minimizes a Euclidean distance between third boundary region 3D coordinate values calculated by a depth value in the third boundary region and fourth boundary region 3D coordinate values calculated by a depth value in the fourth boundary region, and minimizes the depth difference between the first boundary region and the second boundary region and the depth difference between the third boundary region and the fourth boundary region by using a final linear function having a coefficient and a constant which are respectively equal to an average of coefficients of the first linear function and the second linear function and an average of constants of the first linear function and the second linear function.

8 . The 3D around view providing apparatus according to claim 7 , wherein
the first boundary region 3D coordinate values are 3D coordinate values set to correspond to a focal length and a principal point of each of the image pickup units, based on the depth value in the first boundary region and first boundary region 2D coordinate values, the second boundary region 3D coordinate values are 3D coordinate values set to correspond to a focal length and a principal point of each of the image pickup units, based on the depth value in the second boundary region and second boundary region 2D coordinate values, the third boundary region 3D coordinate values are 3D coordinate values set to correspond to a focal length and a principal point of each of the image pickup units, based on the depth value in the third boundary region and third boundary region 2D coordinate values, and the fourth boundary region 3D coordinate values are 3D coordinate values set to correspond to a focal length and a principal point of each of the image pickup units, based on the depth value in the fourth boundary region and fourth boundary region 2D coordinate values.

9 . The 3D around view providing apparatus according to claim 2 , further comprising:
an operator configured to control an operation of an autonomous driving mode of the vehicle; and a communicator configured to receive DNN data based on a downlink (DL) grant of a 5th-generation (5G) network connected to allow the vehicle to operate in the autonomous driving mode, wherein the depth estimator updates the DNN encoder based on the DNN data received from the communicator.

10 . A three-dimensional (3D) around view providing method of providing a 3D around view through a user interface module included in a vehicle, the 3D around view providing method comprising:
receiving a plurality of images through an image pickup unit; acquiring a plurality of depth maps corresponding to the plurality of images; minimizing a depth difference between a first boundary region of a first depth map and a second boundary region of a second depth map, the first boundary region and the second boundary region being boundary regions in which the first depth map and the second depth map, which are adjacent to each other among the plurality of depth maps, overlap; acquiring a first around view depth map with the minimized depth difference between the first boundary region and the second boundary region; restoring a 3D around view image for the plurality of images by using the first around view depth map; and displaying the 3D around view image.

11 . The 3D around view providing method according to claim 10 , wherein
the acquiring the plurality of depth maps comprises acquiring the plurality of depth maps corresponding to the plurality of images by inputting the plurality of images to a deep neural network (DNN) enco

Assignees

• Lg Electronics Inc

Inventors

• Seo Jeong Hyeon
• Kim Sang Hoon
• Im Jin Seok

Classifications

• H04W72/23

  in the downlink direction of a wireless link, i.e. towards a terminal · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• H04N13/156 Primary

  Mixing image signals · CPC title