Computer vision method and system

Patent metadata

Field	Value
Publication number	US-11568595-B2
Application number	US-202117179747-A
Country	US
Kind code	B2
Filing date	Feb 19, 2021
Priority date	Aug 11, 2020
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

A computer vision method for generating a three dimensional reconstruction of an object, the method comprising:
receiving a set of photometric stereo images of the object, the set of photometric stereo images comprising a plurality of images using illumination from different directions using one or more light sources; using a trained neural network to generate a normal map of the object; and producing a 3D reconstruction of said object from said normal map, wherein using said trained neural network comprises converting said set of photometric stereo images to an input form suitable for an input layer of said neural network, wherein said input form comprises, for each pixel, a representation of the different lighting directions and their corresponding intensities which have been obtained from photometric stereo images to which a compensation has been applied, the compensation being determined from an estimate of the distance between the lighting source and a point on the object to which the pixel corresponds.

First claim

The invention claimed is:

1. A computer vision method comprising:
receiving a set of photometric stereo images of an object, the set of photometric stereo images comprising a plurality of images using illumination from different lighting directions using one or more light sources;
using a trained neural network to generate a normal map of the object; and
producing a three-dimensional (3D) reconstruction of the object from the normal map,
wherein using the trained neural network comprises converting the set of photometric stereo images to an input form suitable for an input layer of the trained neural network,
wherein the input form comprises, for each pixel, a representation of the different lighting directions and their corresponding intensities which have been obtained from the photometric stereo images to which a compensation has been applied, the compensation being determined from an estimate of a distance between a light source of the light sources and a point on the object to which the pixel corresponds, and
wherein the input form comprises an observation map provided for each pixel, the observation map including channels which correspond to components of a viewing vector.

2. The computer vision method of claim 1 , wherein, once the normal map is generated by the trained neural network, the 3D reconstruction of the object is obtained and the obtained 3D reconstruction of the object is then used to produce an updated input form and a further normal map is generated from the updated input form.

3. The computer vision method of claim 2 , wherein generating the normal map is an iterative process where successive reconstructions of the object are used to produce updated input forms until convergence of the reconstruction of the object, the updated input forms including the updated input form.

4. The computer vision method of claim 3 ,
wherein the input form comprises a plurality of observation maps, and
wherein an observation map is provided for each pixel, the observation map comprising a projection of the lighting directions onto a 2D plane, the lighting directions for each pixel being derived from a compensated photometric stereo image of the photometric stereo images to which the compensation has been applied.

5. The computer vision method of claim 4 , wherein the observation maps of the plurality of observation maps are updated by compensating the photometric stereo images using a latest reconstruction of the object.

6. The computer vision method of claim 5 , wherein the photometric stereo images are compensated for real-world illumination effects, a compensation amount being derived from the latest reconstruction of the object.

7. The computer vision method of claim 6 , wherein the compensation amount is based on 3D coordinates of a point on the latest reconstruction of the object.

8. The computer vision method of claim 6 , wherein the compensation amount is provided by:

α
m

⁡

(
X
)

=

∅
m

⁢

(

L
^

m

⁡

(
X
)

·

S
^

m

)

μ
m



L
m

⁡

(
X
)



2

where α m (X) is an attenuation factor, X is 3D surface point coordinates of a point, the point coordinates expressed as X=[x, y, z] T , Ø m is intrinsic brightness of the light source m, S m is a principle direction of the light source, μ m is an angular dissipation factor, and {circumflex over (L)} m (X) is a lighting direction, the lighting direction defined as L m (X)/∥L m (X)∥.

9. The computer vision method of claim 4 , wherein the observation map for each pixel is a three-dimensional map comprising a plurality of channels, the channels corresponding to components of a viewing vector.

10. The computer vision method of claim 9 , wherein at least one of the plurality of channels of the observation map comprises information from which the viewing vector for each pixel can be derived.

11. The computer vision method of claim 9 , wherein the plurality of channels of the observation maps comprise at least one of viewing vector components {circumflex over (V)} x or {circumflex over (V)} y .

12. A non-transitory carrier medium carrying computer readable instructions adapted to cause a computer to perform the method of claim 1 .

13. A computer implemented method of training a model,
wherein the model comprises:
a conversion stage to convert a set of photometric stereo images to an input form suitable for an input of the model, the input form comprising, for each pixel, a representation of different lighting directions and their corresponding intensities which have been obtained from the photometric stereo images to which a compensation has been applied, the compensation being determined from an estimate of a distance between a light source and a point on an object to which the pixel corresponds;
a first processing stage to process the input form with a neural network processing stage to produce a normal map of the object; and
a second processing stage to process the normal map of the object to produce an output three dimensional (3D) reconstruction of the object,

wherein the set of photometric stereo images comprises a plurality of images using illumination from different directions using one or more light sources,
wherein the method comprises:
providing training data, the training data comprising the set of photometric stereo images of the object and 3D reconstruction of the object; and
training the model using the photometric stereo images as the input and comparing an output of the trained model with the 3D reconstruction, and

wherein the input form comprises an observation map provided for each pixel, the observation map including channels which correspond to components of a viewing vector.

14. The computer implemented method of claim 13 ,
wherein the input form comprises a plurality of observation maps, and
wherein an o

Assignees

• Toshiba Kk

Inventors

• Logothetis Fotios
• Mecca Roberto
• Budvytis Ignas
• Cipolla Roberto

Classifications

• G06T15/506

  Illumination models · CPC title

• G06T7/00

  Image analysis · CPC title

• G06N3/08

  Learning methods · CPC title

• G06T15/503

  Blending, e.g. for anti-aliasing · CPC title

• G06T7/586 Primary

  from multiple light sources, e.g. photometric stereo · CPC title