Cloud enabled augmented reality

Patent metadata

Field	Value
Publication number	US-10885714-B2
Application number	US-201816029530-A
Country	US
Kind code	B2
Filing date	Jul 6, 2018
Priority date	Jul 7, 2017
Publication date	Jan 5, 2021
Grant date	Jan 5, 2021

Abstract

An augmented reality system generates computer-mediated reality on a client device. The client device has sensors including a camera configured to capture image data of an environment and a location sensor to capture location data describing a geolocation of the client device. The client device creates a three-dimensional (3-D) map with the image data and the location data for use in generating virtual objects to augment reality. The client device transmits the created 3-D map to an external server that may utilize the 3-D map to update a world map stored on the external server. The external server sends a local portion of the world map to the client device. The client device determines a distance between the client device and a mapping point to generate a computer-mediated reality image at the mapping point to be displayed on the client device.

First claim

The invention claimed is:

1. A method of generating computer mediated reality data on a client device, the method comprising:
capturing image data with a camera integrated in the client device, the image data representing a near real-time view of an environment around the client device;
capturing location data with a location sensor integrated in the client device, the location data describing a spatial position of the client device in the environment;
generating local map data based on the image data and the location data, the local map data including one or more three-dimensional (3D) point clouds spatially describing one or more objects in the environment around the client device;
accessing a set of viewpoints stored locally on the client device based on the spatial position of the client device;
identifying a first image in the image data as a novel viewpoint based at least in part on the first image having less than a minimum overlap in field of view with each viewpoint in the set of viewpoints;
transmitting the local map data and the first image as the novel viewpoint to an external server;
receiving a local portion of world map data at the client device from the external server, wherein the world map data is generated by fusing a plurality of local maps generated by one or more client devices and viewpoints captured by one or more client devices, wherein the local portion is selected based on the transmitted local map data;
determining a distance between a mapping point in the local portion of world map data and the spatial position of the client device in the local portion of world map data based on location data;
generating a computer mediated reality image at the mapping point in the local portion of world map data based on the image data, the location data, and the distance between the mapping point and the spatial position of the client device; and
displaying the computer mediated reality image at the mapping point.

2. The method of claim 1 , further comprising transmitting the image data to the external server, wherein the local portion of world map data is selected further based on the image data.

3. The method of claim 1 , further comprising:
identifying a second image in the image data as a second novel viewpoint based at least in part on the second image having less than the minimum overlap in field of view with the first set of viewpoints and the first image; and
transmitting the second image as the second novel viewpoint to the external server.

4. The method of claim 1 , wherein the computer mediated reality image comprises a virtual object that is fixed at the mapping point in the local portion of world map data.

5. The method of claim 4 , further comprising receiving a virtual object from the external server based on the image data and the location data, wherein the computer mediated reality image comprises the received virtual object.

6. The method of claim 1 , wherein generating local map data based on the image data and the location data further comprises:
identifying one or more objects in the environment based on the image data;
determining one or more spatial positions for each of the objects based on the image data and the location data; and
generating a 3D point cloud comprising a set of 3-D points for each of the objects.

7. The method of claim 6 , wherein generating local map data based on the image data and the location data further comprises:
classifying each object into one of a plurality of object types, the plurality of objects types including a stationary type describing objects that are expected to remain in substantially a same spatial position.

8. The method of claim 1 , further comprising:
capturing subsequent location data with the location sensor describing a subsequent spatial position of the client device in the environment;
determining a subsequent distance between the mapping point in the local portion of world map data and the subsequent spatial position of the client device based on location data and the local portion of world map data;
adjusting the computer mediated reality image at the mapping point in the local portion of world map data based on the subsequent location data; and
displaying the adjusted computer mediated reality image at the mapping point.

9. A method comprising:
storing world map data that describes an aggregate of a plurality of environments at a plurality of geolocations and viewpoint data describing images captured at the plurality of environments;
receiving, from a client device, local map data, image data of an environment captured by a camera on the client device, location data describing a spatial position of the client device captured with a location sensor, and one or more novel viewpoints, wherein the local map data is generated by the client device based on the image data and the location data and includes one or more 3D point clouds spatially describing one or more objects in the environment around the client device;
identifying a set of viewpoints from the viewpoint data based on the location data;
for each novel viewpoint, determining whether the novel viewpoint has less than a minimum overlap in field of view with each viewpoint in the set of viewpoints;
responsive to determining that a first novel viewpoint has less than the minimum overlap in field of view with each viewpoint in the set of viewpoints, adding the first novel viewpoint to the set of viewpoints;
retrieving a local portion of world map data based on the location data and the local map data;
updating the local portion of the world map data with the local map data by adding one or more of the 3D point clouds in the local map data to the world map data; and
transmitting the local portion of the world map data to the client device.

10. The method of claim 9 , further comprising:
generating a virtual object to be displayed by the client device in the environment based on the image data and the location data; and
transmitting the virtual object to the client device.

11. The method of claim 9 , further comprising:
determining, for each novel viewpoint, whether a subset of the set of viewpoints having more than the minimum overlap in field of view of the novel viewpoint was captured over a threshold interval of time the novel viewpoint; and
responsive to determining that a first subset of the set of viewpoints having more than the minimum overlap in field of view of a second novel viewpoint was captured over the threshold interval of time from the second novel viewpoint, adding the second novel viewpoint to the set of viewpoints.

12. A non-transitory computer-readable storage medium storing instructions for generating computer mediated reality data on a client device, the instructions, that when executed by a processor, cause the processor to perform operations comprising:
capturing image data with a camera integrated in the client device, the image data representing a near real-time view of an environment around the client device;
capturing location data with a location sensor integrated in the client device, the location data describing a spatial position of the client device in the environment;
generating local map data based on the image data and the location data, the local map data including one or more three-dimensional (3D) point clouds spatially describing one or more objects in the environment around the client device;
accessing a set of viewpoints stored locally on the client device based on the spatial position of the client device;
identifying a first image in the image data as a novel viewpoint based at least in part on the first image having less than a minimum overlap in fie

Assignees

• Niantic Inc

Inventors

• Finman Ross Edward
• Hu Si Ying Diana

Classifications

• G06T2219/2004

  Aligning objects, relative positioning of parts · CPC title

• G06T2200/08

  involving all processing steps from image acquisition to 3D model generation · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• H04N7/18

  Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast · CPC title

• G06V10/751

  Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching · CPC title