Methods and systems for light field augmented reality/virtual reality on mobile devices

Patent metadata

Field	Value
Publication number	US-10388069-B2
Application number	US-201514848759-A
Country	US
Kind code	B2
Filing date	Sep 9, 2015
Priority date	Sep 9, 2015
Publication date	Aug 20, 2019
Grant date	Aug 20, 2019

Abstract

Methods and apparatus are described that enable augmented or virtual reality based on a light field. A geometric proxy of a mobile device such as a smart phone is used during the process of inserting a virtual object from the light field into the real world images being acquired. For example, a mobile device includes a processor and a camera coupled to the processor. The processor is configured to define a view-dependent geometric proxy, record images with the camera to produce recorded frames and, based on the view-dependent geometric proxy, render the recorded frames with an inserted light field virtual object.

First claim

What is claimed:

1. A mobile device, comprising:
a processor;
a camera coupled to the processor; and
a storage device configured to store a plurality of light field views of a physical object;
wherein the processor is configured to:
select a subset of the plurality of light field views based on a user-specified distance from the mobile device;
define a view-dependent geometric proxy at the user-specified distance from the mobile device, wherein an orientation of the view-dependent geometric proxy is based on a location and an orientation angle of the camera;
transform each light field view from each of the selected subset of the light field views based on an intrinsic camera matrix defined for the mobile device, a three dimensional (3D) orientation angle associated with each light field view, a 3D location associated with each light field view, a 3D orientation angle of the mobile device, a 3D position of the mobile device, and a position and orientation of the view-dependent geometric proxy to produce transformed light field views;
interpolate the transformed light field views to produce a light field virtual object;
record images with the camera to produce recorded frames; and
render the recorded frames with the light field virtual object that is adjusted based on the view-dependent geometric proxy as the mobile device moves.

2. The mobile device of claim 1 , wherein the processor determines a new subset of light field views to select based on movement of the mobile device in 3D space.

3. The mobile device of claim 1 , wherein the processor is further configured to:
transform a mapping function for each of the selected subset of the light field views based on the view-dependent geometric proxy to produce transformed mapping functions;
interpolate the transformed mapping functions to produce an interpolated mapping function; and
render the recorded frames with the light field virtual object using the interpolated mapping function.

4. The mobile device of claim 1 , further comprising an inertial measurement unit (IMU), wherein the processor determines the view-dependent geometric proxy based, in part, on sensor signals from the IMU.

5. The mobile device of claim 1 , further comprising a touch sensitive display, wherein the processor receives a signal from the touch sensitive display indicating the user-specified distance from the mobile device as to where the light field virtual object is to appear to be located upon its insertion into the recorded frames captured by the mobile device.

6. The mobile device of claim 1 , wherein the mobile device is a phone or tablet device.

7. A non-transitory storage device including instructions which, when executed by a processor, cause the processor to:
store a plurality of light field views of a physical object;
select a subset of the plurality of light field views based on a user-specified distance from a mobile device;
determine a view-dependent geometric proxy at the user-specified distance as the mobile device moves in three dimensional (3D) space, wherein an orientation of the view-dependent geometric proxy is based on a location and an orientation angle of a camera;
transform each light field view from each of the selected subset of the light field views based on an intrinsic camera matrix defined for the non-transitory storage device, a 3D orientation angle associated with each light field view, a 3D location associated with each light field view, a 3D orientation angle of the non-transitory storage device, a 3D position of the non-transitory storage device, and a position and orientation of the view-dependent geometric proxy to produce transformed light field views;
interpolate the transformed light field views to produce a light field virtual object;
receive video frames recorded by the camera; and
render the video frames on a display with the light field virtual object that is adjusted based on the view-dependent geometric proxy as the mobile device moves.

8. The non-transitory storage device of claim 7 , wherein the subset of light field views are selected from the plurality of light field views between locations at which the light field views are acquired and a current location of the camera.

9. The non-transitory storage device of claim 8 , wherein the instructions, when executed by the processor, cause the processor to transform the subset of light field views based, at least in part, on a geometrical relationship between a position and the orientation of the geometric proxy and a position and orientation associated with the subset of light field views.

10. The non-transitory storage device of claim 7 , wherein the instructions, when executed by the processor, cause the processor to receive an input from a user input device indicating a location in the video frames to insert the light field virtual object.

11. The non-transitory storage device of claim 7 , wherein the instructions, when executed by the processor, cause the processor to determine the view-dependent geometric proxy based on an inertial simultaneous localization and mapping technique.

12. An augmented or virtual reality method implemented at a mobile device, comprising:
storing a plurality of light field views of a physical object;
selecting a subset of the plurality of light field views based on a user-specified distance from a mobile device;
receiving, via a user interface, an input as to a location to insert a light field object in images captured by a mobile camera;
recording video with the mobile camera to produce recorded video frames;
defining a view-dependent geometric proxy for the mobile camera at a user-specified distance from the mobile camera, wherein an orientation of the view-dependent geometric proxy is based on a location and an orientation angle of the mobile camera;
transforming each light field view from each of the selected subset of the light field views based on an intrinsic camera matrix defined for the mobile device, a three dimensional (3D) orientation angle associated with each light field view, a 3D location associated with each light field view, a 3D orientation angle of the mobile device, a 3D position of the mobile device, and a position and orientation of the view-dependent geometric proxy to produce transformed light field views;
interpolating the transformed light field views to produce a light field virtual object; and
rendering the recorded video frames with the light field virtual object based on the view-dependent geometric proxy, wherein the rendering of the recorded video frames with the light field virtual object is adjusted as the mobile camera moves.

13. The method of claim 12 , wherein rendering the recorded video frames with the inserted light field object includes:
determining a position and rotation of the mobile camera;
computing a transformation of each selected light field view of the selected subset of the light field views based on the view-dependent geometric proxy to produce a computed transformation for each of the selected light field views of the subset of the light field views;
remapping each of the subset of light field views using the computed transformation for each such selected light field view to produce remapped selected light field views;
generating a remapped nearby view using the remapped selected light field views; and
embedding the remapped nearby view into the recorded video frames.

14. The method of claim 13 , further comprising:
determining a mapping function for each frame of each selected light field view;
remapping each mapping function using the computed transformation to produce a transformed mapping fun

Assignees

• Futurewei Technologies Inc

Inventors

• Gu Jinwei
• Wilburn Bennett
• Jiang Wei

Classifications

• G06T2200/21

  involving computational photography · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title