Hybrid 2D/3D data in a virtual environment

Patent metadata

Field	Value
Publication number	US-10438413-B2
Application number	US-201715806125-A
Country	US
Kind code	B2
Filing date	Nov 7, 2017
Priority date	Nov 7, 2017
Publication date	Oct 8, 2019
Grant date	Oct 8, 2019

Abstract

A method for using a virtual reality (VR) headset to view a two-dimensional (2D) technical drawing of a physical object of a real-world, real-world environment in three dimensions (3D), the method comprising: using LiDAR to produce a 3D point cloud of the real-world environment; scaling and aligning the 2D technical drawing to match the size and orientation of the physical object as depicted in the 3D point cloud; overlaying the 2D technical drawing (including all labels and dimensions) over the physical object as depicted in the 3D point cloud; and visually comparing the 3D point cloud representation of the physical object to the 2D technical drawing by simultaneously displaying the 3D point cloud of the real-world environment and the overlaid 2D technical drawing to a user with the VR headset.

First claim

We claim:

1. A method for using a virtual reality (VR) headset to view a two-dimensional (2D) technical drawing of a real-world, physical object in three dimensions (3D), the method comprising:
using LiDAR to produce a 3D point cloud of a real-world environment that comprises the physical object by performing the following substeps:
scanning the real-world environment with a laser scanner to produce scan data;
registering the scan data with a computer to stitch data from multiple scans together to create a data-intensive point cloud;
degrading the data-intensive point cloud by reducing the number of data points in the data-intensive point cloud to create a degraded point cloud by using an octree filter to subdivide the data-intensive point cloud and preserving only one point from each nonempty subdivision;
creating the 3D point cloud by formatting the degraded point cloud into a game-engine-compatible format; and
importing the 3D point cloud into a software framework used for creating and developing video games;

scaling and aligning the 2D technical drawing to match the size and orientation of the physical object as depicted in the 3D point cloud;
overlaying the 2D technical drawing (including all labels and dimensions) on the physical object depicted in the 3D point cloud; and
visually comparing the 3D point cloud representation of the physical object to the 2D technical drawing by simultaneously displaying the 3D point cloud of the real-world environment and the overlaid 2D technical drawing to a user with the VR headset.

2. The method of claim 1 , wherein the game-engine-compatible format is ASCII.

3. The method of claim 1 , wherein the game-engine-compatible format is polygonal file format (PLY).

4. The method of claim 1 , wherein the degraded point cloud comprises fewer than two million data points.

5. The method of claim 1 , further comprising removing a background from the 2D technical drawing leaving only lines, labels, and dimensions such that when the 3D point cloud of the real-world environment and the overlaid 2D technical drawing are simultaneously displayed to the user with the VR headset the user only sees the lines, labels, and dimensions of the 2D technical drawing overlaid on the physical object as depicted in the 3D point cloud.

6. The method of claim 1 , wherein when the 3D point cloud of the real-world environment and the overlaid 2D technical drawing are simultaneously displayed to the user with the VR headset the physical object as depicted in the 3D point cloud is obscured from the view of the user by the overlaid 2D technical drawing.

7. The method of claim 1 , wherein the physical object is a floor and the 2D technical drawing is a floor plan.

8. The method of claim 1 , wherein the physical object is a wall.

9. The method of claim 1 , wherein the 2D technical drawing pre-exists the production of the 3D point cloud.

10. A method for using a virtual reality (VR) headset to view a two-dimensional (2D) technical drawing of a real-world physical object in three dimensions (3D), the method comprising:
scanning a real-world environment that comprises the physical object to produce scan data;
creating with a computer a 3D point cloud of the real-world environment based on the scan data comprising the substeps of:
registering the scan data with the computer such that scan data from multiple scans are stitched together to create a data-intensive point cloud;
degrading the data-intensive point cloud by reducing the number of data points in the data-intensive point cloud to create a degraded point cloud by using an octree filter to subdivide the data-intensive point cloud and preserving only one point from each nonempty subdivision;
creating the 3D point cloud by formatting the degraded point cloud into a game-engine-compatible format; and
importing the 3D point cloud into a software framework used for creating and developing video games;

overlaying the 2D technical drawing (including all lines, labels, and dimensions) on the physical object as depicted in the 3D point cloud; and
visually comparing the 3D point cloud representation of the physical object to the 2D technical drawing by simultaneously displaying the 3D point cloud of the real-world environment and the overlaid 2D technical drawing to a user with the VR headset.

11. The method of claim 10 , wherein prior to the overlaying step, the 2D technical drawing is scaled and aligned to match the size and orientation of the physical object as depicted in the 3D point cloud.

12. The method of claim 11 , wherein the scanning step is performed with a LiDAR laser scanner.

13. The method of claim 10 , wherein a background of the overlaid 2D technical drawing is sufficiently transparent such that when the 3D point cloud of the real-world environment and the overlaid 2D technical drawing are simultaneously displayed to the user with the VR headset the lines, labels, and dimensions of the overlaid 2D technical drawing as well as the physical object as depicted in the 3D point cloud are visible to the user in the VR headset.

Assignees

• Spawar Systems Ct Pacific
• Us Navy

Inventors

• Bilinski Mark
• Greunke Larry Clay

Classifications

• G01S17/89

  for mapping or imaging · CPC title

• G01S17/00

  Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems · CPC title

• H04N13/344

  with head-mounted left-right displays · CPC title

• G06T2210/04

  Architectural design, interior design · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title