Information processing apparatus, control method for same, non-transitory computer-readable storage medium, and vehicle driving support system

Patent metadata

Field	Value
Publication number	US-12008778-B2
Application number	US-202016921977-A
Country	US
Kind code	B2
Filing date	Jul 7, 2020
Priority date	Jan 15, 2018
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

This invention provides an information processing apparatus comprising a first acquiring unit configured to acquire first distance information from a distance sensor, a second acquiring unit configured to acquire an image from an image capturing device, a holding unit configured to hold a learning model for estimating distance information from images, an estimating unit configured to estimate second distance information corresponding to the image acquired by the second acquiring unit using the learning model, and a generating unit configured to generate third distance information based on the first distance information and the second distance information.

First claim

The invention claimed is:

1. An information processing apparatus comprising:
(A) one or more circuits, or
(B) at least one processor and at least one memory, or
(C) at least one processor, at least one memory, and one or more circuits,
wherein the (A) one or more circuits, (B) at least one processor and at least one memory, or (C) at least one processor, at least one memory, and one more circuits is configured to function as a plurality of units comprising:
(1) a first acquiring unit configured to acquire first distance image information comprising distance as a pixel value using a distance sensor;
(2) a second acquiring unit configured to acquire an image captured by an image capturing device;
(3) an estimating unit configured to estimate second distance image information comprising distance as a pixel value from the image acquired by the second acquiring unit using a learning model which has been generated so as to output distance image information corresponding to an input image; and
(4) a calculation unit configured to calculate third distance image information comprising distance as a pixel value in accordance with (a) the first distance image information and (b) the second distance image information.

2. The information processing apparatus according to claim 1 , wherein the plurality of units further comprises:
a map holding unit configured to hold, as a three-dimensional map, a position and orientation of the image capturing device and distance image information corresponding to the position and orientation;
a position-and-orientation-calculation unit configured to calculate a position and orientation of the image capturing device based on the three-dimensional map and the image acquired by the second acquiring unit;
an updating unit configured to update the three-dimensional map based on the position and orientation calculated by the position-and-orientation-calculation unit and the second distance image information estimated by the estimating unit; and
a degree-of-reliability-determining unit configured to determine a degree of reliability of the first distance image information and the distance image information in the three-dimensional map, and
wherein the calculation unit calculates the third distance image information based on (1) the first distance image information, (2) the distance image information in the three-dimensional map updated by the updating unit, and (3) the degree of reliability.

3. The information processing apparatus according to claim 2 , wherein the degree-of-reliability-determining unit determines the degree of reliability based on whether or not the first distance image information is present.

4. The information processing apparatus according to claim 2 , wherein the distance sensor is a LIDAR-based sensor, and
wherein the degree-of-reliability-determining unit determines a degree of reliability of the first distance image information in accordance with the intensity of laser light reflected from a target object.

5. The information processing apparatus according to claim 2 , wherein the degree-of-reliability-determining unit determines a degree of reliability of the first distance image information based on an operating distance of a sensor.

6. The information processing apparatus according to claim 2 , wherein the degree-of-reliability-determining unit determines the degree of reliability based on continuity of distance information.

7. The information processing apparatus according to claim 2 , wherein the distance sensor is a stereo-type distance sensor, and
wherein the degree-of-reliability-determining unit performs the determination based on a degree of similarity in stereo matching performed when the distance sensor acquires distance image information.

8. The information processing apparatus according to claim 2 , wherein the plurality of units further comprises a semantic region segmenting unit configured to segment the image acquired by the second acquiring unit into semantic regions, and
wherein the calculation unit combines the first distance image information and the distance image information in the three-dimensional map updated by the updating unit based on the semantic regions.

9. The information processing apparatus according to claim 1 , wherein a position and an orientation of the information processing apparatus are estimated using the third distance image information.

10. The information processing apparatus according to claim 1 , wherein the plurality of units further comprises:
an environment recognizing unit configured to recognize a surrounding environment based on the third distance image information and the image acquired by the second acquiring unit; and
a position-and-orientation-acquiring unit configured to acquire a position and an orientation of the apparatus relative to the environment.

11. A vehicle driving support system comprising: the distance sensor; the image capturing device; and the information processing apparatus according to claim 2 .

12. The vehicle driving support system according to claim 11 , further comprising an actuator for controlling the movement of a vehicle, wherein the plurality of units further comprises a driving-control unit configured to control the actuator based on (1) the surrounding environment, (2) the position, and (3) the orientation.

13. A control method for an information processing apparatus comprising a holding unit which holds a learning model for estimating distance information from images, the control method comprising:
acquiring first distance image information comprising distance as a pixel value using a distance sensor;
acquiring an image captured by an image capturing device;
estimating second distance image information comprising distance as a pixel value from the acquired image using a learning model which has been generated so as to output distance image information corresponding to an input image; and
calculating third distance image information comprising distance as a pixel value in accordance with (a) the first distance image information and (b) the second distance image information.

14. A non-transitory computer-readable storage medium storing a program which causes, when read and executed by a computer, the computer to execute a control method for an information processing apparatus comprising a holding unit which holds a learning model for estimating distance information from images, the control method comprising:
acquiring first distance image information comprising distance as a pixel value using a distance sensor;
acquiring an image captured by an image capturing device;
estimating second distance image information comprising distance as a pixel value from the acquired image using a learning model which has been generated so as to output distance image information corresponding to an input image; and
calculating third distance image information comprising distance as a pixel value in accordance with (a) the first distance image information and (b) the second distance image information.

15. The information processing apparatus according to claim 1 , wherein the plurality of units further comprises a degree-of-reliability-determining unit configured to determine a first degree of reliability of the first distance image information and a second degree of reliability of the second distance image information, and
wherein the calculation unit calculates the third distance image information by performing weighted averaging of the first distance image information and the second distance image information using the first degree of reliability and the second degree of reliabilit

Assignees

• Canon Kk

Inventors

• Kasuya Nozomu
• Kobayashi Kazuhiko
• Suzuki Masahiro
• Kotake Daisuke
• Tomioka Makoto

Classifications

• G05D1/2435

  Extracting 3D information · CPC title

• G08G1/16

  Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08) · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10012

  Stereo images · CPC title