Point cloud data processing apparatus, point cloud data processing method, and program

Patent metadata

Field	Value
Publication number	US-12277741-B2
Application number	US-202117496078-A
Country	US
Kind code	B2
Filing date	Oct 7, 2021
Priority date	Jun 14, 2019
Publication date	Apr 15, 2025
Grant date	Apr 15, 2025

Abstract

A point cloud data processing apparatus includes: an image data acquisition unit that acquires image data of an object; a point cloud data acquisition unit that acquires point cloud data; a recognition unit that recognizes the object on the basis of the image data, and acquires a region of the object and attribute information for identifying the object; and an attribute assigning unit that selects, from the point cloud data, point cloud data that belongs to the region of the object, and assigns the identified attribute information to the selected point cloud data.

First claim

What is claimed is:

1. A point cloud data processing apparatus comprising:
a processor that is configured to
acquire image data of an object,
acquire point cloud data having a corresponding positional relationship with the image data and representing pieces of three-dimensional information of a large number of points on an outer surface of the object,
recognize the object on the basis of the image data, and acquire a region of the object and attribute information for identifying the object,
select, from the point cloud data, point cloud data that belongs to the region of the object, and assign the acquired attribute information to the selected point cloud data,
recognize a missing part of the point cloud data that belongs to the region of the object on the basis of the region of the object and the attribute information,
calculate point cloud data of the missing part by linear interpolation using point cloud data in a neighborhood or by spline interpolation, and
determine whether the missing part exists or not in the point cloud data on the basis of luminance of the image data,

wherein the processor is further configured to
project three-dimensional coordinates of each point onto a specific plane to generate the image data having pixel values in accordance with reflection intensity or distance of each point of the point cloud data, and
assign the attribute information based on the generated image data,

wherein the processor is configured to
group the point cloud data for each partial region of the object by
recognizing, in a case where the image data includes image data of the object having a plurality of partial regions, a region of each partial region among the plurality of partial regions, and acquiring the region of the each partial region and attribute information for identifying the each partial region, and
selecting, from the point cloud data, point cloud data that belongs to the region of each partial region among the partial regions on a per partial region basis, and assigning the attribute information of the partial region to the point cloud data selected on a per partial region basis, and

wherein the processor is configured to identify each partial region of the object by image processing for extracting contours of the object to acquire the attribute information of each partial region.

2. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to
acquire, in a case where the image data includes pieces of image data of a plurality of objects, a region of each object among the objects and attribute information for identifying the object on a per object basis, and
select, from the point cloud data, point cloud data that belongs to the region of each object among the objects on a per object basis, and assign the attribute information of the object to the point cloud data selected on a per object basis.

3. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to
assign the attribute information to the point cloud data of the missing part.

4. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to acquire a captured image acquired by image capturing of the object as the image data.

5. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to
generate the image data on the basis of the point cloud data, and
acquire the generated image data.

6. The point cloud data processing apparatus according to claim 1 , wherein the processor is formed of a recognizer subjected to machine learning or deep learning.

7. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to generate a three-dimensional model of the object on the basis of the point cloud data that is assigned the attribute information.

8. The point cloud data processing apparatus according to claim 1 , wherein the image data and the point cloud data are respectively acquired by devices having a common optical axis.

9. The point cloud data processing apparatus according to claim 1 , wherein the image data and the point cloud data are respectively acquired by devices for which a positional relationship between the devices is known.

10. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to acquire data obtained by a laser scanner.

11. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to acquire the point cloud data by using a Time-of-Flight (ToF) camera or a stereo camera.

12. The point cloud data processing apparatus according to claim 1 , wherein the processor is configured to identify each partial region of the object by a recognizer constructed through machine learning or deep learning to acquire the attribute information of each partial region.

13. A point cloud data processing method comprising:
acquiring image data of an object;
acquiring point cloud data having a corresponding positional relationship with the image data and representing pieces of three-dimensional information of a large number of points on an outer surface of the object;
recognizing the object on the basis of the image data, and acquiring a region of the object and attribute information for identifying the object;
selecting, from the point cloud data, point cloud data that belongs to the region of the object, and assigning the acquired attribute information to the selected point cloud data;
recognizing a missing part of the point cloud data that belongs to the region of the object on the basis of the region of the object and the attribute information for identifying the object;
calculating point cloud data of the missing part by linear interpolation using point cloud data in a neighborhood or by spline interpolation; and
determining whether the missing part exists or not in the point cloud data on the basis of luminance of the image data,
wherein the method further comprises:
projecting three-dimensional coordinates of each point onto a specific plane to generate the image data having pixel values in accordance with reflection intensity or distance of each point of the point cloud data, and
assigning the attribute information based on the generated image data,

wherein the method further comprises:
grouping the point cloud data for each partial region of the object by
recognizing, in a case where the image data includes image data of the object having a plurality of partial regions, a region of each partial region among the plurality of partial regions, and acquiring the region of the each partial region and attribute information for identifying the each partial region, and
selecting, from the point cloud data, point cloud data that belongs to the region of each partial region among the partial regions on a per partial region basis, and assigning the attribute information of the partial region to the point cloud data selected on a per partial region basis, and

wherein the method further comprises identifying each partial region of the object by image processing for extracting contours of the object to acquire the attribute information of each partial region.

14. A non-transitory computer readable recording medium storing a program for causing a computer to perform a point cloud data process comprising:
acquiring image data of an object;
acquiring point cloud data having a corresponding positional relationship with the image data and representing pieces of three-dimensional information of a large number of points on an outer surfac

Assignees

• Fujifilm Corp

Inventors

• Iwami Kazuchika

Classifications

• G06F18/21

  Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T17/00

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06V20/00

  Scenes; Scene-specific elements (control of digital cameras H04N23/60) · CPC title