Mobile augmented reality system

Patent metadata

Field	Value
Publication number	US-9600933-B2
Application number	US-201113175608-A
Country	US
Kind code	B2
Filing date	Jul 1, 2011
Priority date	Jul 1, 2011
Publication date	Mar 21, 2017
Grant date	Mar 21, 2017

Abstract

Embodiments of the invention relate to systems, apparatuses and methods to provide image data, augmented with related data, to be displayed on a mobile computing device. Embodiments of the invention display a live view augmented with information identifying an object amongst other objects. Embodiments of the invention may utilize other related data, such as 3D point cloud data, image data and location data related to the object, to obtain a specific location of an object within the live view. Embodiments of the invention may further display a live view with augmented data three-dimensionally consistent with the position and orientation of the image sensor of the mobile computing device.

First claim

The invention claimed is:

1. A method comprising:
detecting an object in a live view image captured by an image sensor;
generating a three-dimensional (3D) model generated for the object;
determining a position and orientation of the image sensor by identifying segments of the object in the live view image;
extracting visual features of the object using the position and orientation of the image sensor;
generating a first two-dimensional (2D) projection mask corresponding to a first user selected point of interest (POI) included in the 3D model;
generating a second 2D projection mask corresponding to a second user selected point POI included in the 3D model; and
augmenting the live view image with the extracted visual features of the object and the first and second projection masks to display the first and second POIs within the object.

2. The method of claim 1 , further comprising processing a database of 3D cloud data for the object.

3. The method of claim 2 , wherein processing the 3D cloud data comprises extracting the 2D planar segments from the 3D cloud data.

4. The method of claim 3 , wherein processing the 3D cloud data further comprises:
combining sets of point cloud data; and
associating the point cloud data with location data.

5. The method of claim 4 , wherein processing the 3D cloud data further comprises processing 2D images of the object from multiple angles with corresponding point cloud data sets from multiple angles and view positions.

6. The method of claim 5 , further comprising receiving information regarding the reliability of the 3D cloud data.

7. The method of claim 1 , wherein determining the position and orientation of the image sensor further comprises:
matching the visible object segments with object segments of the 3D model.

8. The method of claim 1 , wherein each of the one or more projection masks are geometrically consistent with a surface plane of the object.

9. The method of claim 1 , wherein the object comprises a building and the detected surface of the structure comprises an external side of the building.

10. An apparatus comprising:
a processor;
a memory; and
an augmentation module included in the memory and executed via the processor to:
detect an object in a live view image captured by an image sensor;
generate a three-dimensional (3D) model generated for the object;
determine a position and orientation of the image sensor by identifying segments of the object in the live view image;
extract visual features of the object using the position and orientation of the image sensor;
generate a first two-dimensional (2D) projection mask corresponding to a first user selected point of interest (POI) included in the 3D model;
generate a second 2D projection mask corresponding to a second user selected point POI included in the 3D model; and
augment the live view image with the extracted visual features of the object and the first and second projection masks to display the first and second POIs within the object.

11. The apparatus of claim 10 , further wherein the augmentation module processes a database of 3D cloud data for the object.

12. The method of claim 11 , wherein processing the 3D cloud data comprises extracting the 2D planar segments from the 3D cloud data.

13. The method of claim 12 , wherein processing the 3D cloud data further comprises:
combining sets of point cloud data; and
associating the point cloud data with location data.

14. The method of claim 13 , wherein processing the 3D cloud data further comprises processing 2D images of the object from multiple angles with corresponding point cloud data sets from multiple angles and view positions.

15. The method of claim 14 , further comprising receiving information regarding the reliability of the 3D cloud data.

16. The method of claim 10 , wherein determining the position and orientation of the image sensor comprises:
identifying visible object segments in the live view image; and
matching the visible object segments with object segments of the 3D model.

17. The method of claim 10 , wherein each of the one or more projection masks are geometrically consistent with a surface plane of the object.

18. The apparatus of claim 10 , wherein the object comprises a building and the detected surface of the structure comprises an external side of the building.

19. The apparatus of claim 10 , wherein the apparatus comprises a mobile computing device, the mobile computing device to further include a display to display an augmented live view including the image data associated with the POI.

20. An article of manufacture comprising a non-transitory machine-readable storage medium that provides instructions that, if executed by the machine, will cause the machine to perform operations comprising:
detecting an object in a live view image captured by an image sensor;
generating a three-dimensional (3D) model generated for the object;
determining a position and orientation of the image sensor by identifying segments of the object in the live view image;
extracting visual features of the object using the position and orientation of the image sensor;
generating a first two-dimensional (2D) projection mask corresponding to a first user selected point of interest (POI) included in the 3D model;
generating a second 2D projection mask corresponding to a second user selected point POI included in the 3D model; and
augmenting the live view image with the extracted visual features of the object and the first and second projection masks to display the first and second POIs within the object.

21. The article of manufacture of claim 20 , further comprising processing a database of 3D cloud data for the object.

22. The article of manufacture of claim 21 , wherein processing the 3D cloud data comprises extracting the 2D planar segments from the 3D cloud data.

23. The article of manufacture of claim 22 , wherein processing the 3D cloud data further comprises:
combining sets of point cloud data; and
associating the point cloud data with location data.

24. The article of manufacture of claim 23 , wherein processing the 3D cloud data further comprises processing 2D images of the object from multiple angles with corresponding point cloud data sets from multiple angles and view positions.

25. The article of manufacture of claim 24 , further comprising receiving information regarding the reliability of the 3D cloud data.

26. The article of manufacture of claim 20 , wherein determining the position and orientation of the image sensor comprises:
matching the visible object segments with object segments of the 3D model.

Assignees

• Wu Yi
• Takacs Gabriel
• El Choubassi Maha
• Kozintsev Igor V
• Intel Corp

Inventors

• Wu Yi
• Takacs Gabriel
• El Choubassi Maha
• Kozintsev Igor V

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title