Method for 3d reconstruction of an environment of a mobile device, corresponding computer program product and device

Patent metadata

Field	Value
Publication number	US-2018160102-A1
Application number	US-201715829171-A
Country	US
Kind code	A1
Filing date	Dec 1, 2017
Priority date	Dec 1, 2016
Publication date	Jun 7, 2018
Grant date	—

Abstract

A method is proposed for 3D reconstruction of an environment of a mobile device comprising a camera. The method includes calculating a coarse 3D reconstruction of at least one of the environment by a first reconstruction method that takes into account first pictures of the at least one area captured by the camera, determining if at least one target part exists in the environment based on a detection of at least one object attribute taking into account at least one of the first pictures, calculating a refined 3D reconstruction of the at least one target part by a second reconstruction method that takes into account second pictures of the at least one target part captured by the camera, and aggregating the calculated reconstructions for providing the 3D reconstructionof the environment.

First claim

1 . A method for 3D reconstruction of an environment of a mobile device comprising at least one camera,
wherein it comprises:
calculating a coarse 3D reconstruction of at least one area of said environment by a first reconstruction method,
said first reconstruction method taking into account at least first pictures of said at least one area captured by said at least one camera;
determining automatically if at least one target part exists in said environment based on at least a detection of at least one object attribute,
said detection taking into account at least one of said first pictures;
calculating a refined 3D reconstruction of said at least one target part by a second reconstruction method,
said second reconstruction method taking into account at least second pictures of said at least one target part captured by said at least one camera;
aggregating the calculated reconstructions for providing said 3D reconstruction of said environment.

2 . The method according to claim 1 , wherein said at least one object attribute belongs to the group comprising:
a saliency attribute representative of a quality by which said target part stands out relative to its neighborhood; a geometry attribute of said target part; a category attribute representative of an object classification of said target part; and a weighted combination of said saliency attribute, said geometry attribute, and said category attribute.

3 . The method according to claim 2 , wherein said at least one geometry attribute belongs to the group comprising:
a scale size; a distribution density of 3D points; a planarity; and a shape.

4 . The method according to claims 1 , wherein said determining automatically further comprises:
localizing at least one localized area in said environment through a user interface of said mobile device;
said at least one target part being determined automatically in said at least one localized area.

5 . The method according to claims 1 , wherein said calculating a refined 3D reconstruction of said at least one target part further comprises:
validating said at least one target part by a user of said mobile device;
said calculating a refined 3D reconstruction being performed when said at least one target part is validated.

6 . The method according to claims 1 , wherein said calculating a coarse 3D reconstruction of at least one area of said environment further comprises:
activating said at least one camera in a first mode of operation for capturing said first pictures.

7 . The method according to claims 1 , wherein said first reconstruction method belongs to the group comprising:
Structure from Motion SfM; Multi-View Stereo MVS; and Simultaneous Localization And Mapping SLAM.

8 . The method according to claims 1 , wherein said mobile device further comprises a depth sensor, and wherein said coarse 3D reconstruction of at least one area of said environment further takes into account depth maps of said area delivered by said depth sensor.

9 . The method according to claims 1 , and wherein said calculating said refined 3D reconstruction of said at least one target part further comprises:
activating said at least one camera in a second mode of operation for capturing said second pictures.

10 . The method according to claim 9 , wherein said mobile device further comprises at least one flash light, wherein said at least one flash light is activated in said second mode, and wherein said calculating said refined 3D reconstruction of said at least one target part enforces a multiview photometric stereo method taking into account photometric data based on said second pictures and on an associated position of said at least one flash light, said associated position of said at least one flash light being estimated from a position of said at least one camera of said mobile device.

11 . The method according to claim 10 , wherein said at least one object attribute comprises a category representative of an object classification of said at least one target part, and said multiview photometric stereo method further takes into account a reflectance associated with said object classification of said at least one target part.

12 . The method according to claims 1 , wherein said aggregating the reconstructions calculated for said at least one area enforces a multi-view stereo methodology for providing a multi-resolution representation as being said 3D reconstruction of said environment.

13 . A device for 3D reconstruction of an environment of a mobile device comprising at least one camera, wherein said device comprises:
a memory; and at least one processor configured for: calculating a coarse 3D reconstruction of at least one area of said environment by a first reconstruction method, said first reconstruction method taking into account at least first pictures of said at least one area captured by said at least one camera; determining automatically if at least one target part exists in said environment based on at least a detection of at least one object attribute, said detection taking into account at least one of said first pictures; calculating a refined 3D reconstruction of said at least one target part by a second reconstruction method, said second reconstruction method taking into account at least second pictures of said at least one target part captured by said at least one camera; aggregating the calculated reconstructions for providing said 3D reconstruction of said environment.

14 . The device according to claim 13 wherein said at least one processor is further configured for calculating said refined 3D reconstruction of said at least one target part by:
activating said at least one camera in a second mode of operation for capturing said second pictures.

15 . A mobile device comprising a device according to claim 13 , said mobile device being preferably chosen among a mobile phone and a tablet.

16 . The device according to claim 14 wherein said mobile device further comparising at least one flash light, said at least one flash light is activated in said second mode,
and wherein said at least one processor is further configured for calculating said refined 3D reconstruction of said at least one target part by enforcing a multiview photometric stereo method taking into account photometric data based on said second pictures and on an associated position of said at least one flash light, said associated position of said at least one flash light being estimated from a position of said at least one camera of said mobile device.

17 . The device according to claim 16 wherein said at least one object attribute comprises a category representative of an object classification of said at least one target part, and said multiview photometric stereo method further takes into account a reflectance associated with said object classification of said at least one target part.

18 . The device according to claim 13 wherein said at least one processor is further configured for determining automatically if said at least one target part exists in said environment based on at least a detection of said at least one object attribute by:
localizing at least one localized area in said environment through a user interface of said mobile device; said at least one target part being determined automatically in said at least one localized area.

19 . The device according to claim 13 wherein said at least one object attribute belongs to the group comprising:
a saliency attribute representative of

Assignees

• Thomson Licensing

Inventors

• Luo Tao
• Robert Philippe
• Alleaume Vincent

Classifications

• H04N13/254 Primary

  in combination with electromagnetic radiation sources for illuminating objects · CPC title

• G06T7/55 Primary

  from multiple images · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title