System and process for projecting location-referenced panoramic images into a 3-D environment model and rendering panoramic images from arbitrary viewpoints within the 3-D environment model

Patent metadata

Field	Value
Publication number	US-9369679-B2
Application number	US-83722407-A
Country	US
Kind code	B2
Filing date	Aug 10, 2007
Priority date	Nov 7, 2006
Publication date	Jun 14, 2016
Grant date	Jun 14, 2016

Abstract

A system and process of morphing location-referenced panoramic images into views at nearby locations. When using panoramic images in an interactive tour, a user might desire to see the environment from viewpoints for which no panoramic images are available. This invention makes this possible. It enables a user to experience views from arbitrary locations in the environment, so as long as one or more panoramic images are available at nearby locations. In particular, this invention makes it possible to combine two non-overlapping geo-referenced panoramic video streams into a new video stream which seamlessly transitions between these streams. When used in a client-server architecture, this invention also makes it possible for the server to transmit a sparse sequence of panoramic images, and provide the user with a dense panoramic video stream, by synthesizing the missing panoramic images. Said system and process is also applicable to incomplete panoramic images, photographs, and video.

First claim

The invention claimed is:

1. An image acquisition system comprising:
a camera for acquiring panoramic images;
a sensor for generating a 3-D environment model;
a positioning system;
a computer system, where the computer system is configured by a first server application and a second server application; and
a panoramic image database;
where the computer system is configured by the first server application to:
receive a plurality of panoramic images captured by the camera;
receive a plurality of locations captured by the positioning system corresponding to positions where the plurality of panoramic images are captured;
generate a plurality of annotated panoramic images by associating the plurality of panoramic images with the corresponding plurality of locations; and
upload the plurality of annotated panoramic images to the image database; and

where the computer system is configured by the second server application to:
receive a user-selected viewing location;
determine a virtual location corresponding to the user-selected viewing location in the 3-D environment model;
select a subset of panoramic images from the plurality of annotated panoramic images having an annotated location near the virtual location in the 3-D environment model;
project the subset of panoramic images into the 3-D environment model at the virtual location;
blend the projected subset of panoramic images to create a new panoramic image.

2. The image acquisition system of claim 1 , wherein the positioning system further comprises a global positioning system and an inertial measurement unit.

3. The image acquisition system of claim 1 , wherein the sensor for generating the 3-D environment model is a laser range finder.

4. The image acquisition system of claim 3 , further configured to:
acquire environment data using the laser range finder; and
build the 3-D environment model using the environment data.

5. The image acquisition system of claim 1 , wherein the 3-D environment model is a sphere.

6. The image acquisition system of claim 1 , wherein the plurality of panoramic images are from a video.

7. The image acquisition system of claim 1 , wherein the computer system is further configured to blend the projected subset of panoramic images to create a new panoramic image by weighted blending.

8. The image acquisition system of claim 1 , wherein the camera for acquiring panoramic images is a plurality of cameras with partially overlapping fields.

9. The image acquisition system of claim 1 , wherein the computer system further comprises a user computer system configured to execute a client application in a web browser application.

10. The image acquisition system of claim 1 , wherein the computer system further comprises a first computer to utilize the first server application and a second computer to utilize the second server application.

Assignees

• Arfvidsson Joakim
• Thrun Sebastian
• Univ Leland Stanford Junior

Inventors

• Arfvidsson Joakim
• Thrun Sebastian

Classifications

• H04N7/181 Primary

  for receiving images from a plurality of remote sources · CPC title

• G06T15/205

  Image-based rendering · CPC title

• G06T3/0062

  Physics · mapped topic

• G06T3/12

  Panospheric to cylindrical image transformations · CPC title