Generating immersive trip photograph visualizations

Patent metadata

Field	Value
Publication number	US-10825246-B2
Application number	US-201816144487-A
Country	US
Kind code	B2
Filing date	Sep 27, 2018
Priority date	Sep 27, 2018
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

A user selects a set of photographs from a trip through an environment that he or she desires to present to other people. A collection of photographs, including the set of photographs captured during the trip optionally augmented with additional photographs obtained from another collection, are combined with a terrain model (e.g., a digital elevation model) to extract information regarding the geographic location of each of the photographs within the environment. The collection of photographs are analyzed, considering their geolocation information as well as the photograph content to register the photographs relative to one another. This information for the photographs is compared to the terrain model in order to accurately position the viewpoint for each photograph within the environment. A presentation of the selected photographs within the environment is generated that displays both the selected photographs and synthetic data filled in beyond the edges of the selected photographs.

First claim

What is claimed is:

1. In a digital medium generation environment, a method implemented by at least one computing device, the method comprising:
obtaining a set of photographs taken on a trip in which a user followed a path through a geographical area;
receiving a three dimensional terrain model of the geographical area;
receiving an indication of the geographical area where the set of photographs were captured;
determining both a location and an orientation of ones of the set of photographs within the three dimensional terrain model of the geographical area;
receiving an indication of user-selected photographs of the set of photographs; and
generating a fly-through view through the three dimensional terrain model, the fly-through view along the path through the geographical area, the fly-through view comprising a sequence of images along the path displayed as a video, the sequence of images including each of the user-selected photographs as well as, for each of the user-selected photographs, data from the three dimensional terrain model that fills in data surrounding the user-selected photograph.

2. The method as described in claim 1 , wherein the determining comprises:
determining whether the set of photographs includes at least a threshold number of photographs that capture a particular area on the trip; and
augmenting the set of photographs with one or more additional photographs in response to the set of photographs not including at least the threshold number of photographs that capture the particular area on the trip, but excluding the one or more additional photographs from the fly-through view.

3. The method as described in claim 1 , wherein the determining comprises:
extracting visual features from each photograph in the set of photographs;
determining, based on the visual features, both three dimensional location information and orientation information for each photograph in the set of photographs;
reconstructing terrain represented in the set of photographs by generating a point cloud from the set of photographs; and
aligning the reconstructed terrain with the three dimensional terrain model to fine-tune the three dimensional location information and the orientation information for each photograph in the set of photographs.

4. The method as described in claim 3 , wherein the reconstructing comprises using a structure from motion pipeline to generate the point cloud.

5. The method as described in claim 1 , wherein receiving the indication of the geographical area comprises receiving a selection of the geographic area from the user.

6. The method as described in claim 1 , wherein generating the fly-through view comprises determining the path through the geographical area based on locations where the user-selected photographs were captured.

7. The method as described in claim 6 , wherein determining the path comprises using a Catmull-Rom spline with the locations of each of the user-selected photographs as control points.

8. The method as described in claim 1 , wherein generating the fly-through view comprises using as the path through the geographical area an actual path followed by the user based on GPS location data obtained during the trip.

9. The method as described in claim 1 , wherein generating the fly-through view comprises determining, as the fly-through view progresses from a first location on the path where a first photograph of the set of photographs was captured to a second location on the path where a second photograph of the set of photographs was captured, a camera orientation by interpolating the camera orientation associated with the photograph captured at the first location and the camera orientation associated with the photograph captured at the second location.

10. The method as described in claim 1 , wherein the fly-through view comprises images generated from the three dimensional terrain model between locations along the path where user-selected photographs were captured.

11. The method as described in claim 10 , wherein the fly-through view comprises a passive fly-through view in which at each location where a user-selected photograph was captured, the user-selected photograph is displayed.

12. The method as described in claim 10 , wherein the fly-through view comprises an interactive fly-through view in which at each location where a user-selected photograph was captured, the user-selected photograph is displayed and a viewer can look around the location in a 360 degree view.

13. In a digital medium generation environment, a device comprising:
one or more processors;
one or more computer-readable storage media having stored thereon multiple instructions that, in response to execution by the one or more processors, cause the one or more processors to:
obtain a set of photographs taken on a trip in which a user followed a path through a geographical area;
receive a three dimensional terrain model of the geographical area;
receive an indication of the geographical area where the set of photographs were captured;
determine both a location and an orientation of ones of the set of photographs within the three dimensional terrain model of the geographical area;
receive an indication of user-selected photographs of the set of photographs; and
generate a fly-through view through the three dimensional terrain model, the fly-through view along the path through the geographical area, the fly-through view comprising a sequence of images along the path displayed as a video including each of the user-selected photographs as well as, between the user-selected photographs, images generated from the three dimensional terrain model.

14. The device as described in claim 13 , wherein the multiple instructions that cause the one or more processors to determine both the location and the orientation of ones of the set of photographs comprise instructions that cause the one or more processors to:
extract visual features from each photograph in the set of photographs;
determine, based on the visual features, both three dimensional location information and orientation information for each photograph in the set of photographs;
reconstruct terrain represented in the set of photographs by generating a point cloud from the set of photographs; and
align the reconstructed terrain with the three dimensional terrain model to fine-tune the dimensional location information and the orientation information for each photograph in the set of photographs.

15. The device as described in claim 13 , wherein the fly-through view comprises a passive fly-through view in which at each location where a user-selected photograph was captured, the user-selected photograph is displayed and an area surrounding the user-selected photograph is filled in with data from the three dimensional terrain model.

16. The device as described in claim 13 , wherein the fly-through view comprises an interactive fly-through view in which at each location where a user-selected photograph was captured, the user-selected photograph is displayed and a viewer can look around the location in a 360 degree view.

17. In a digital medium generation environment, a system comprising:
means for obtaining a set of photographs taken on a trip in which a user followed a path through a geographical area;
means for receiving a three dimensional terrain model of the geographical area;
means for receiving an indication of the geographical area where the set of photographs were captured;
means for determining both a location and an orientation of ones of the set of photographs within the three dimension

Assignees

• Adobe Inc
• Brno Univ Of Technology

Inventors

• Luká{Hacek Over (U)} Michal
• Chen Zhili
• Brejcha Jan
• Cadik Martin

Classifications

• G06V20/70

  Labelling scene content, e.g. deriving syntactic or semantic representations · CPC title

• G06V20/38

  Outdoor scenes · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06F16/29

  Geographical information databases · CPC title

• G06T19/003 Primary

  Navigation within 3D models or images · CPC title