System and method for three-dimensional scanning and for capturing a bidirectional reflectance distribution function

Patent metadata

Field	Value
Publication number	US-10055882-B2
Application number	US-201715678075-A
Country	US
Kind code	B2
Filing date	Aug 15, 2017
Priority date	Aug 15, 2016
Publication date	Aug 21, 2018
Grant date	Aug 21, 2018

Abstract

A method for generating a three-dimensional (3D) model of an object includes: capturing images of the object from a plurality of viewpoints, the images including color images; generating a 3D model of the object from the images, the 3D model including a plurality of planar patches; for each patch of the planar patches: mapping image regions of the images to the patch, each image region including at least one color vector; and computing, for each patch, at least one minimal color vector among the color vectors of the image regions mapped to the patch; generating a diffuse component of a bidirectional reflectance distribution function (BRDF) for each patch of planar patches of the 3D model in accordance with the at least one minimal color vector computed for each patch; and outputting the 3D model with the BRDF for each patch.

First claim

What is claimed is:

1. A method for generating a three-dimensional (3D) model of an object, comprising:
capturing a plurality of images of the object from a plurality of viewpoints, the images comprising a plurality of color images;
generating a 3D model of the object from the images, the 3D model comprising a plurality of planar patches;
for each patch of the planar patches:
mapping a plurality of image regions of the plurality of images to the patch, each image region comprising at least one color vector; and
computing, for each patch, at least one minimal color vector among the color vectors of the image regions mapped to the patch;

generating a diffuse component of a bidirectional reflectance distribution function (BRDF) for each patch of planar patches of the 3D model in accordance with the at least one minimal color vector computed for each patch;
outputting the 3D model with the BRDF for each patch, the BRDF further comprising a specular component separate from the diffuse component;
rendering one or more diffuse views of the object;
computing a plurality of features based on the one or more diffuse views of the object; and
assigning a classification to the object in accordance with the plurality of features, the classification comprising one of: a defective classification and a clean classification,
wherein the assigning the classification to the object in accordance with the plurality of features is performed by a convolutional neural network, and
wherein the convolutional neural network is trained by:
receiving a plurality of training 3D models of objects and corresponding training classifications;
rendering a plurality of views of the 3D models with controlled lighting to generate training data;
computing a plurality of feature vectors from the views by the convolutional neural network;
computing parameters of the convolutional neural network;
computing a training error metric between the training classifications of the training 3D models with outputs of the convolutional neural network configured based on the parameters;
computing a validation error metric in accordance with a plurality of validation 3D models separate from the training 3D models;
in response to determining that the training error metric and the validation error metric fail to satisfy a threshold, rendering additional views of the 3D models with different controlled lighting to generate additional training data;
in response to determining that the training error metric and the validation error metric satisfy the threshold, configuring the neural network in accordance with the parameters;
receiving a plurality of test 3D models of objects with unknown classifications;
rendering a plurality of views of the test 3D models with controlled lighting to generate testing data; and
classifying the test 3D models using the rendered views of the test 3D models and the configured convolutional neural network.

2. The method of claim 1 , further comprising:
aligning the 3D model with a reference model;
comparing the 3D model to the reference model to compute a plurality of differences between corresponding portions of the 3D model and the reference model; and
detecting a defect in the object when one or more of the plurality of differences exceeds a threshold.

3. The method claim 1 , further comprising: receiving a user input specifying one or more parameters of the specular component of the BRDF.

4. The method of claim 3 , wherein the specified one or more parameters of the specular component of the BRDF are applied to a selected portion of the 3D model.

5. The method of claim 1 , further comprising computing the specular component of the BRDF, the computing the specular component comprising, for each of the planar patches:
subtracting the at least one minimal color vector from the color vector of each of the image regions mapped to the patch to compute a plurality of specular images of the patch; and
computing one or more parameters of the specular component.

6. The method of claim 5 , wherein the computing the one or more parameters of the specular component comprises:
initializing the one or more parameters;
rendering the 3D model in accordance with the BRDF set in accordance with the one or more parameters to render a plurality of rendered views of the patch;
computing an error function in accordance with a difference between the rendered views of the patch with the image regions mapped to the patch; and
computing the one or more parameters by iteratively updating the one or more parameters to minimize the error function.

7. The method of claim 1 , further comprising:
arranging the 3D model in a virtual environment including a virtual camera and at least one light source;
rendering an image of the virtual environment including the 3D model, the image comprising at least one specular highlight from the reflection of the at least one light source off the 3D model; and
displaying the image.

8. The method of claim 1 , wherein the plurality of images are captured by a plurality of different cameras.

9. The method of claim 8 , wherein the plurality of images of the object are captured while the object is on a conveyor belt.

10. The method of claim 1 , wherein the plurality of images of the object from the plurality of viewpoints are captured by a single camera.

11. The method of claim 10 , wherein the camera is a stereoscopic depth camera comprising a first infrared camera, a second infrared camera, and a color camera.

12. The method of claim 1 , wherein the 3D model is a model of less than the entire exterior surface of the object.

13. A system for generating a three-dimensional (3D) model of an object, the system comprising:
a depth camera system;
a processor coupled to the depth camera system; and
memory having instructions stored thereon that, when executed by the processor, cause the processor to:
capture a plurality of images of the object from a plurality of viewpoints, the images comprising a plurality of color images;
generate a 3D model of the object from the images, the 3D model comprising a plurality of planar patches;
for each patch of the planar patches:
map a plurality of image regions of the plurality of images to the patch, each image region comprising at least one color vector; and
compute, for each patch, at least one minimal color vector among the color vectors of the image regions mapped to the patch;

generate a diffuse component of a bidirectional reflectance distribution function (BRDF) for each patch of planar patches of the 3D model in accordance with the at least one minimal color vector computed for each patch;
output the 3D model with the BRDF for each patch, the BRDF further comprising a specular component separate from the diffuse component;
render one or more diffuse views of the object;
compute a plurality of features based on the one or more diffuse views of the object; and
assign a classification to the object in accordance with the plurality of features, the classification comprising one of: a defective classification and a clean classification,

wherein the instructions configured to cause the processor to assign the classification to the object in accordance with the plurality of features supply the features to a convolutional neural network, and
wherein the convolutional neural network is trained by:
receiving a plurality of training 3D models of objects and corresponding training classifications;
rendering a plurality of views of the 3D models with controlled lighting to generate training data;
computing a plurality of feature vectors from the views by the convolutional neural

Assignees

• Aquifi Inc

Inventors

• Marin Giulio
• Rafii Abbas
• Dal Mutto Carlo
• Tieu Kinh
• Murali Giridhar
• Memo Alvise

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/75

  Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06T11/10

  Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77) · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title