3d video reconstruction system

Patent metadata

Field	Value
Publication number	US-2017193686-A1
Application number	US-201615394740-A
Country	US
Kind code	A1
Filing date	Dec 29, 2016
Priority date	Dec 30, 2015
Publication date	Jul 6, 2017
Grant date	—

Abstract

A server receives video data and location data from mobile devices. Each mobile device records a video of a target. The location data identifies a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device to the target. The location data is associated with a corresponding video frame from the video data. The server identifies video frames from the video data captured from the mobile devices. The server scales parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target. The server extracts the scaled parts of the identified video frames and generates a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices.

First claim

What is claimed is:

1 . A method comprising:
receiving, at a server, video data and location data from a plurality of mobile devices, each mobile device configured to record a video of a target, the location data identifying a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device to the target, the location data associated with a corresponding video frame from the video data; identifying video frames from the video data captured from the plurality of mobile devices; scaling parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target; extracting the scaled parts of the identified video frames; and generating a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices.

2 . The method of claim 1 , further comprising:
forming an augmented reality content comprising the three-dimensional model of the target; associating the augmented reality content with an identifier of a physical object; and communicating the augmented reality content to an augmented reality viewing device, the augmented reality viewing device configured to display the three-dimensional model in response to detecting the identifier of the physical object in a field of view of the augmented reality viewing device.

3 . The method of claim 2 , further comprising:
identifying a missing portion on the three-dimensional model of the target; identifying a first head mounted display based on a location of the missing portion on the three-dimensional model relative to a first position and a first orientation of the first head mounted display; generating a first augmented reality content for the first head mounted display, the first augmented reality content including a suggested virtual path to guide the first head mounted display to a second position and a second orientation of the first head mounted display, the first head mounted display configured to generate video data corresponding to the missing portion on the three-dimensional model while at the second position and the second orientation; and communicating the first augmented reality content to the first head mounted display.

4 . The method of claim 3 , wherein the first mobile device includes a transparent display configured to display the suggested virtual path in the transparent display relative to a position of the target within a field of view of the transparent display.

5 . The method of claim 2 , further comprising:
receiving a request from the augmented reality viewing device, the request identifying a side view of the target; identifying a first mobile device based on its location and position and the requested side view of the target; generating and communicating instructions to the first mobile device, the instructions identifying a direction in which the mobile device is to move to generate the requested side view of the target.

6 . The method of claim 1 , further comprising:
identifying a first video frame of a first mobile device and a second video frame of a second mobile device, both first and second video frames captured at a same time; extracting a first region in the first video frame and a second region in the second video frame; scaling the first region of the first video frame based on the position and distance of the first mobile device, and the second region of the second video frame based on the position and distance of the second mobile device; determining whether a portion of the first scaled region and the second region overlap and contain a substantially similar pixel pattern; and including the first scaled and second scaled region in a portion of the three-dimensional model of the target in response to determining that the first portion of the first scaled region and the second region overlap and contain the substantially similar pixel pattern.

7 . The method of claim 1 , wherein each mobile device comprises:
a transparent display configured to display an augmented reality content identifying the target; a camera configured to capture video frames of the target; an inertial navigation sensor configured to determine a position and an orientation of the mobile device; a location sensor configured to determine a geographic location of the mobile device and detect a frame of reference at a geographic location of the target; and a processor configured to generate video data and associate a video frame of the video data with a corresponding location and position of the mobile device relative to the target and the frame of reference, the location data including the location and position of the mobile device relative to the target and the frame of reference.

8 . The method of claim 1 , wherein each mobile device comprises:
a transparent display configured to display an augmented reality content identifying the target; a camera configured to capture video frames of the target; an inertial navigation sensor configured to determine a position and an orientation of the mobile device; a depth sensor configured to determine a depth of the target relative to the mobile device; a location sensor configured to identify a geographic location of the mobile device and detect a frame of reference at a geographic location of the target; and a processor configured to generate video data and associate a video frame of the video data with a corresponding location and position of the mobile device relative to the target, the location data including the position and orientation of the mobile device relative to the target, and the depth of the target relative to the mobile device.

9 . The method of claim 1 , further comprising:
generating an augmented reality content corresponding to the target, the augmented reality content identifying a physical object as the target in the plurality of mobile devices; and communicating the augmented reality content to the plurality of mobile devices, each mobile device configured to display the augmented reality content as a layer on top of the physical object in a corresponding field of view of the mobile device.

10 . The method of claim 1 , wherein the plurality of mobile devices includes:
a first mobile device having a static location and forming a frame of reference with the target; and a second mobile device moving along an undetermined path, a location of the second mobile device determined relative to the frame of reference.

11 . A server comprising:
a processor; and a memory storing instructions that, when executed by the processor, configure the server to:
receive video data and location data from a plurality of mobile devices, each mobile device configured to record a video of a target, the location data identifying a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device and the target, the location data associated with a corresponding video frame from the video data;
identify video frames from the video data captured from the plurality of mobile devices;
scale parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target;
extract the scaled parts of the identified video frames; and
generate a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices.

12 . The server of claim 11 , wherein the instructions further configure the server to:
form an augmented reality content comprising the three-dimensional model of t

Assignees

• Daqri Llc

Inventors

• Mullins Brian

Classifications

• G06T3/40 Primary

  Scaling of whole images or parts thereof, e.g. expanding or contracting · CPC title

• G06T19/003

  Navigation within 3D models or images · CPC title

• G02B2027/0141

  characterised by the informative content of the display · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06T15/205 Primary

  Image-based rendering · CPC title