4D vizualization of building design and construction modeling with photographs

Patent metadata

Field	Value
Publication number	US-9852238-B2
Application number	US-201514690104-A
Country	US
Kind code	B2
Filing date	Apr 17, 2015
Priority date	Apr 24, 2014
Publication date	Dec 26, 2017
Grant date	Dec 26, 2017

Abstract

A system and method are disclosed for, using structure-from-motion techniques, projecting a building information model (BIM) into images from photographs taken of a construction site, to generate a 3D point cloud model using the BIM and, when combined with scheduling constraints, facilitates 4D visualizations and progress monitoring. One of the images acts as an anchor image. Indications are received of first points in the anchor image that correspond to second points in the BIM. Calibration information for an anchor camera is calculated based on the indications and on metadata extracted from the anchor image, to register the anchor image in relation to the BIM. A homography transformation is determined between the images and the anchor camera using the calibration information, to register the rest of the images with the BIM, where some of those images are taken from different cameras and from different angles to the construction site.

First claim

What is claimed is:

1. A system comprising:
at least one processing device;
non-transitory computer-readable medium storing a three-dimensional (“3D”) building information model (BIM), a first plurality of images of a construction site, and instructions, the at least one processing device to execute the instructions to:
receive an anchor image chosen from the first plurality of images;
receive indications of a plurality of first points in the anchor image that correspond to a plurality of second points in the BIM;
calculate camera calibration information for an anchor camera from which the anchor image was taken based on the indications and on metadata extracted from the anchor image, to register the anchor image in relation to the BIM, and thus generate a registered anchored image; and
determine a homography transformation between the first plurality of images and the anchor camera using the camera calibration information for the anchor camera, to register a second plurality of images selected from the first plurality of images other than the anchor image,
wherein some of the second plurality of images come from at least one additional camera and from an angle different than that of the anchor camera.

2. The system of claim 1 , wherein the plurality of first points and the plurality of second points each comprise at least four, wherein the at least one processing device further to construct a visualization of progress of a construction project in relation to the BIM based on the registered anchor image and the registered second plurality of images.

3. The system of claim 1 , wherein the metadata includes a focal length in millimeters, and wherein to calculate the camera calibration information, the at least one processing device further to:
convert, depending on a sensor size of the anchor camera, the focal length to pixels, and thus generate a converted focal length; and
execute a perspective-n-point (PnP) algorithm on the plurality of first points and the plurality of second points, using the converted focal length, to determine a plurality of intrinsic parameters and a plurality of extrinsic parameters of the anchor camera.

4. The system of claim 3 , wherein the plurality of intrinsic parameters comprise (i) x and y coordinates of a center of an image and (ii) skew, wherein the at least one processing device further to execute a three-parameter pin-hole model with variable focal length and two radial distortion coefficients, to fix values of the intrinsic parameters that result in no radial distortion.

5. The system of claim 3 , wherein the plurality of extrinsic parameters comprises rotation and translation.

6. The system of claim 1 , wherein, to register the second plurality of images, the at least one processing device further to:
detect and match interest points across the first plurality of images;
prune the interest points using a Random Sample Consensus (RANSAC) algorithm, to generate useable interest points;
search for a third plurality of images from the second plurality of images having a percentage of inliers within a threshold percentage that include the useable interest points;
warp the first plurality of points of the anchor image to the useable interest points in the third plurality of images containing;
execute a PnP algorithm for each of the third plurality of images using known correspondences between the plurality of first points and the useable interest points, to register nearby images from the third plurality of images; and
calculate camera calibration information for one or more camera from which the third plurality of images are obtained.

7. The system of claim 1 , wherein the anchor image and the second plurality of images comprises registered images, and wherein the at least one processing device further to:
execute constrained bundle adjustment on the registered images by constraining a plurality of three-dimensional points of the registered images to lie within a ray of the anchor camera when triangulated with a feature point from the anchor camera;
identify a third plurality of images from the registered images having a threshold number of features corresponding to a plurality of tracks, wherein each track comprises matched features common to two or more registered images; and
triangulate the plurality of tracks in the registered images, which correspond to the features in the third plurality of images, in relation to 3D feature points in the BIM.

8. The system of claim 7 , wherein the at least one processing device further to:
select an image from the third plurality of images having fewest matches between features above a threshold number of matches; and
register a camera corresponding to the selected image by:
solving a constrained PnP problem using two-dimensional matches corresponding to the 3D feature points that were triangulated; and
making a solution to the constrained PnP problem robust with RANSAC.

9. The system of claim 8 , wherein to improve the solution to the constrained PnP problem the at least one processing device further to:
use a Fundamental matrix between the anchor image and the selected image;
create epipolar lines corresponding to the plurality of first points;
constrain 3D locations within the BIM to lie near the epipolar lines; and
determine a 3D rotation and translation of the selected image that jointly minimizes reprojection error and a point-to-line distance from projected points to their corresponding epipolar lines.

10. The system of claim 7 , wherein the at least one processing device further to:
identify a fourth plurality of images as the registered images other than the third plurality of images;
select an image from the fourth plurality of images with a fewest number of matching tracks;
receive an indication of a plurality of third points in the selected image corresponding to the plurality of second points in the BIM for the anchor image; and
execute a PnP algorithm for the selected image using known correspondences between the plurality of third points and the plurality of second points, to register the selected image with the BIM.

11. The system of claim 10 , wherein the anchor camera comprises a first anchor camera, wherein the at least one processing device further to:
designate a camera from which the selected image is obtained as a second anchor camera; and
execute constrained bundle adjustment on the registered images by constraining a plurality of three-dimensional points of the registered images to lie within a ray of the second anchor camera when triangulated with a feature point from the second anchor camera.

12. A system comprising:
at least one processing device;
non-transitory computer-readable medium storing instructions and a three-dimensional (“3D”) point cloud model of building progress of a construction site comprising time-lapsed pixels obtained from a plurality of images, the at least one processing device to execute the instructions to:
identify a structure within the 3D point cloud model that does not occupy the same pixels in a majority of the time-lapsed pixels over time;
determine a background image within the 3D point cloud model by computing a per-pixel median of the time-lapsed pixels;
compute a squared pixel-wise difference between the structure and the background image, to identify a plurality of occluding pixels of the structure; and
smooth the plurality of occluding pixels with a cross-bilateral filter, to create a smoothed occluded image removable from the 3D point cloud model.

13. The system of claim 12 , wherein the at least one processing device further to threshold the smooth occluded image to e

Assignees

• Univ Illinois

Inventors

• Forsyth David Alexander
• Karsch Kevin
• Golparvar-Fard Mani

Classifications

• G06F30/13 Primary

  Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads · CPC title

• G06F17/5004 Primary

  Physics · mapped topic

• G06T2207/20072

  Graph-based image processing · CPC title

• G06T7/80

  Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration · CPC title

• G06T7/35

  using statistical methods · CPC title