System, device and method of generating a high resolution and high accuracy point cloud

Patent metadata

Field	Value
Publication number	US-11346950-B2
Application number	US-201816195365-A
Country	US
Kind code	B2
Filing date	Nov 19, 2018
Priority date	Nov 19, 2018
Publication date	May 31, 2022
Grant date	May 31, 2022

Abstract

A system, device and method of generating high resolution and high accuracy point cloud. In one aspect, a computer vision system receives a camera point cloud from a camera system and a LiDAR point cloud from a LiDAR system. An error of the camera point cloud is determined using the LiDAR point cloud as a reference. A correction function is determined based on the determined error. A corrected point cloud is generated from the camera point cloud using the correction function. A training error of the corrected point cloud is determined using the first LiDAR point cloud as a reference. The correction function is updated based on the determined training error. When training is completed, the correction function can be used by the computer vision system to generate a generating high resolution and high accuracy point cloud from the camera point cloud provided by the camera system.

First claim

The invention claimed is:

1. A computer vision system, comprising:
a processor system;
a memory coupled to the processor system, the memory tangibly storing thereon executable instructions that, in response to execution by the processor system, cause the computer vision system to:
perform a first training phase in which the computer vision system is caused to:
(i) receive a camera point cloud from a camera system;
(ii) receive a first LiDAR point cloud from a first LiDAR system having a first resolution;
(iii) determine a first error of the camera point cloud using the first LiDAR point cloud as a reference;
(iv) determine a first correction function based on the determined first error;
(v) generate a first corrected point cloud from the camera point cloud using the first correction function;
(vi) determine a first training error of the first corrected point cloud using the first LiDAR point cloud as a reference; and
(vii) update the first correction function based on the determined first training error; and
perform a second training phase subsequent to the first training phase in which the computer vision system is caused to:
(a) receive a second LiDAR point cloud from a second LiDAR system having a second resolution, wherein the second resolution of the second LiDAR system is higher than the first resolution of the first LiDAR system;
(b) determine a second error of the first corrected point cloud using the second LiDAR point cloud as a reference;
(c) determine a second correction function based on the determined second error;
(d) generate a second corrected point cloud using the second correction function;
(e) determine a second training error of the second corrected point cloud using the second LiDAR point cloud as a reference; and
(f) update the second correction function based on the determined second training error.

2. The computer vision system of claim 1 , wherein the executable instructions, in response to execution by the processor system, cause the computer vision system to repeat operations (v)-(vii) until the first training error is less than a first error threshold.

3. The computer vision system of claim 1 , wherein the executable instructions to determine the first error of the camera point cloud using the first LiDAR point cloud as a reference and to determine the first correction function based on the determined first error, in response to execution by the processor system, cause the computer vision system to:
generate a mathematical representation of the determined first error of the camera point cloud using the first LiDAR point cloud as a reference;
determine the first correction function based on the mathematical representation of the determined first error;
wherein the executable instructions to determine the first training error of the first corrected point cloud using the first LiDAR point cloud as a reference and to update the first correction function based on the determined first training error, in response to execution by the processor system, cause the computer vision system to:
generate a mathematical representation of the first training error of the first corrected point cloud using the first LiDAR point cloud as a reference; and
update the first correction function based on the mathematical representation of the first training error.

4. The computer vision system of claim 1 , wherein the executable instructions, in response to execution by the processor system, cause the computer vision system to repeat operations (d)-(f) until the second training error is less than a second error threshold.

5. The computer vision system of claim 1 , wherein the second LiDAR system comprises one or more 64-beam LiDAR units and the first LiDAR system comprises one or more 8-beam, 16-beam or 32-beam LiDAR units.

6. The computer vision system of claim 1 , wherein the camera system is a stereo camera system and the camera point cloud is a stereo camera point cloud.

7. The computer vision system of claim 1 , wherein the processor system comprises a neural network.

8. The computer vision system of claim 1 , wherein the executable instructions to determine the second error of the camera point cloud using the second LiDAR point cloud as a reference and to determine the second correction function based on the determined second error, in response to execution by the processor system, cause the computer vision system to:
generate a mathematical representation of the determined second error of the camera point cloud using the second LiDAR point cloud as a reference;
determine the second correction function based on the mathematical representation of the determined second error;
wherein the executable instructions to determine the second training error of the second corrected point cloud using the second LiDAR point cloud as a reference and to update the second correction function based on the determined second training error, in response to execution by the processor system, cause the computer vision system to:
generate a mathematical representation of the second training error of the second corrected point cloud using the second LiDAR point cloud as a reference; and
update the second correction function based on the mathematical representation of the second training error.

9. The computer vision system of claim 1 , wherein the executable instructions, in response to execution by the processor system, cause the computer vision system to:
(1) calculate an output of a point cloud application algorithm using the first corrected point cloud and an output of the point cloud application algorithm using the first LiDAR point cloud;
(2) determine via a first cost function a first cost of error in the output of the point cloud application algorithm using the first corrected point cloud; and
(3) update the first correction function based on the determined first cost.

10. The computer vision system of claim 9 , wherein the executable instructions, in response to execution by the processor system, cause the computer vision system to repeat operations (1)-(3) until the first cost is less than a first cost threshold.

11. The computer vision system of claim 9 , wherein the executable instructions, in response to execution by the processor system, cause the computer vision system to:
(4) calculate an output of the point cloud application algorithm using the second corrected point cloud and an output of the point cloud application algorithm using the second LiDAR point cloud;
(5) determine via a second cost function a second cost of error in the output of the point cloud application algorithm using the second corrected point cloud; and
(6) update the second correction function based on the determined second cost.

12. The computer vision system of claim 11 , wherein the executable instructions, in response to execution by the processor system, cause the computer vision system to repeat operations (4)-(6) until the second cost is less than a second cost threshold.

13. A computer vision system, comprising:
a processor system;
a memory coupled to the processor system, the memory tangibly storing thereon executable instructions that, in response to execution by the processor system, cause the computer vision system to:
perform a first training phase in which the computer vision system is caused to:
(i) receive a camera point cloud from a camera system;
(ii) receive a first LiDAR point cloud from a first LiDAR system having a first resolution;
(iii) determine a first error of the camera point cloud using the first LiDAR point cloud as a reference;
(iv) determine a first correction function based on the determined first error

Assignees

• Amirloo Abolfathi Elmira
• Golestan Irani Keyvan
• Huawei Tech Co Ltd

Inventors

• Amirloo Abolfathi Elmira
• Golestan Irani Keyvan

Classifications

• G06T7/521

  from laser ranging, e.g. using interferometry; from the projection of structured light · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G01S17/87

  Combinations of systems using electromagnetic waves other than radio waves · CPC title

• G01S17/86 Primary

  Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders · CPC title

• G01S7/4808

  Evaluating distance, position or velocity data · CPC title