Markerless multi-user, multi-object augmented reality on mobile devices

Patent metadata

Field	Value
Publication number	US-9928656-B2
Application number	US-201514852247-A
Country	US
Kind code	B2
Filing date	Sep 11, 2015
Priority date	Sep 11, 2015
Publication date	Mar 27, 2018
Grant date	Mar 27, 2018

Abstract

Methods and devices permit a user to insert multiple virtual objects into a real world video scene. Some inserted objects may be statically tied to the scene, while other objects are designated as moving with certain moving objects in the scene. Markers are not used to insert the virtual objects. Users of separate mobile devices can share their inserted virtual objects to create a multi-user, multi-object augmented reality (AR) experience.

First claim

What is claimed is:

1. A mobile device, comprising:
a processor coupled to a display, wherein the processor is configured to:
generate a first augmented reality scene comprising at least a first virtual object positioned at a first location in a real world scene;
receive data comprising at least a second virtual object from a second mobile device, the received second virtual object having been inserted into the real world scene of a second augmented reality scene on the second mobile device, wherein the data includes location information indicating a second location of the second virtual object within the real world scene in the second augmented reality scene; and
render a shared augmented reality scene on the display, wherein the shared augmented reality scene comprises both the first virtual object positioned at the first location in the real world scene and the received second virtual object positioned at the second location in the real world scene, wherein at least one of the first virtual object and the second virtual object is a dynamic virtual object that is dynamically inserted in the shared augmented reality scene to enable movement of the dynamic virtual object with real moving objects in the real world scene.

2. The mobile device of claim 1 , wherein the processor is configured to:
receive a first input as to 6 degree-of-freedom (DoF) poses to insert a third virtual object into the real world scene; and
render the real world scene on the display with the first virtual object, the second virtual object, and the third virtual object.

3. The mobile device of claim 2 , wherein the processor is configured to:
separate, based on a second input, a video frame of the real world scene into a static portion that contains video content that does not move within the real world scene and a dynamic portion that contains video content that does move within the real world scene, wherein the second input indicates a bounding region for the dynamic portion of the video frame.

4. The mobile device of claim 3 , wherein the first input indicates:
the first virtual object is to be statically tied to the real world scene;
the third virtual object is to be dynamically inserted and move with a movement of the dynamic portion of the real world scene; and
the 6 DoF poses to insert the first and the third virtual objects into the real world scene.

5. The mobile device of claim 1 , wherein the processor is configured to:
extract a set of two-dimensional (2D) keypoints from a plurality of image frames at different points in time;
establish a correspondence between the extracted set of 2D keypoints in the plurality of image frames; and
triangulate corresponding pairs of extracted set of 2D keypoints to determine a location of a three-dimensional (3D) point in 3D space.

6. The mobile device of claim 3 , wherein the processor is configured to compute a current 6 DoF static pose for the mobile device based on the static portion as well as a current 6 DoF dynamic pose for the third virtual object in the dynamic portion.

7. The mobile device of claim 6 , wherein the processor is configured to:
project a priori estimates for a current mobile device pose from a previous filtered 6 DoF static pose and a previous filter 6 DoF dynamic poses; and
compute an improved a posteriori estimate of the current 6 DoF static pose and the current 6 DoF dynamic pose.

8. The mobile device of claim 6 , wherein compute the current 6 DoF dynamic pose for the third virtual object in the dynamic portion comprises:
defining a polyhedron centered at a triangulated dynamic 3D point, wherein the dynamic 3D point being within the dynamic portion of the real world scene;
computing a convex hull region based on projected 2D points of the vertices of each polyhedron; and
averaging the convex hull regions together to obtain a dynamic region of the third virtual object in a current video frame.

9. The mobile device of claim 1 , wherein the processor is configured to
determine a correspondence between 3D points in real world video frames acquired by multiple mobile devices; and
align static 3D points of the multiple mobile devices based upon the determined correspondence, wherein the static 3D points comprise 3D points within a static portion of the real world scene.

10. A method implemented on a mobile device, comprising:
generating a first augmented reality scene comprising at least a first virtual object positioned at a first location in a real world scene;
receiving data comprising at least a second virtual object from a second mobile device, the received second virtual object having been inserted into the real world scene of a second augmented reality scene on the second mobile device, wherein the data includes location information indicating a second location of the second virtual object within the real world scene in the second augmented reality scene; and
rendering a shared augmented reality scene on a display, wherein the shared augmented reality scene comprises both the first virtual object positioned at the first location in the real world scene and the received second virtual object positioned at the second location in the real world scene, wherein at least one of the first virtual object and the second virtual object is a dynamic virtual object that is dynamically inserted in the shared augmented reality scene to enable movement of the dynamic virtual object with real moving objects in the real world scene.

11. The method of claim 10 , further comprising receiving input as to 6 degree-of-freedom (DoF) poses to insert the first virtual object and a third virtual object into the real world scene, wherein rendering comprises rendering the real world scene with the first virtual object, the second virtual object, and the third virtual object.

12. The method of claim 11 , further comprising separating a video frame of the real world scene into a static portion that contains video content that does not move within the real world scene and a dynamic portion that contains video content that does move within the real world scene.

13. The method of claim 12 , wherein the received input indicates that the first virtual object is to be statically inserted into the real world scene, the third virtual object is to be dynamically inserted and move with the dynamic portion of the real world scene, and 6 DoF poses of the first virtual object and third virtual object in the first real world scene.

14. The method of claim 10 , further comprising:
determining a correspondence between static portions in real world video frames acquired by multiple mobile devices; and
aligning static portions of the multiple mobile devices.

Assignees

• Futurewei Technologies Inc

Inventors

• Jiang Wei
• Gu Jinwei

Classifications

• G11B27/00

  Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel · CPC title

• G06T2219/024

  Multi-user, collaborative environment · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06T7/00

  Image analysis · CPC title

• G11B27/031

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title