Synchronized, interactive augmented reality displays for multifunction devices

Patent metadata

Field	Value
Publication number	US-10854008-B2
Application number	US-201916240655-A
Country	US
Kind code	B2
Filing date	Jan 4, 2019
Priority date	Jan 5, 2010
Publication date	Dec 1, 2020
Grant date	Dec 1, 2020

Abstract

A device can receive live video of a real-world, physical environment on a touch sensitive surface. One or more objects can be identified in the live video. An information layer can be generated related to the objects. In some implementations, the information layer can include annotations made by a user through the touch sensitive surface. The information layer and live video can be combined in a display of the device. Data can be received from one or more onboard sensors indicating that the device is in motion. The sensor data can be used to synchronize the live video and the information layer as the perspective of video camera view changes due to the motion. The live video and information layer can be shared with other devices over a communication link.

First claim

What is claimed is:

1. A system, comprising:
one or more sensors;
a camera;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:
generate computer-generated imagery based on video captured from the camera;
determine a current location based on sensor data from the one or more sensors;
establish a communication link with another device;
send, to the other device via the communication link, the video, the computer-generated imagery and information indicating the current location, and
in response to receiving, from the other device, an indication of a location of the other device, update the computer-generated imagery to indicate the location of the other device.

2. The system of claim 1 , further comprising:
a display configured to display the video and the computer-generated imagery, wherein the computer-generated imagery includes an indication of the current location.

3. The system of claim 1 , wherein the memory further comprises instructions that further cause the one or more processors to:
in response to receiving user input indicating one or more annotations, generate additional computer-generated imagery based on the one or more annotations.

4. The system of claim 1 , wherein the memory further comprises instructions that further cause the one or more processors to:
determine a route between the current location and the location of the other device; and
overlay a representation of the route on the computer-generated imagery.

5. The system of claim 1 , wherein the memory further comprises instructions that further cause the one or more processors to:
receive an indication that motion is detected by the one or more sensors according to updated sensor data; and
update the computer-generated imagery according to the updated sensor data.

6. The system of claim 5 , wherein the memory further comprises instructions that further cause the one or more processors to:
send the updated computer-generated imagery to the other device via the communication link.

7. The system of claim 1 , wherein the memory further comprises instructions that further cause the one or more processors to:
identify an object in the video;
generate an information layer including information about the object; and
overlay the information layer onto the computer-generated imagery.

8. A method, comprising:
generating computer-generated imagery based on video captured from a camera of a device;
determining a current location of the device based on sensor data from one or more sensors of the device;
establishing a communication link with another device;
sending, from the device to the other device via the communication link, the video, the computer-generated imagery and information indicating the current location, and
receiving, from the other device, an indication of a location of the other device, wherein the computer-generated imagery indicates the location of the other device.

9. The method of claim 8 , further comprising:
receiving user input indicating one or more annotations; and
generating additional computer-generated imagery based on the one or more annotations.

10. The method of claim 8 , further comprising:
determining a route between the current location and the location of the other device; and
overlaying a representation of the route on the computer-generated imagery.

11. The method of claim 8 , further comprising:
receiving an indication that motion is detected by the one or more sensors according to updated sensor data; and
updating the computer-generated imagery according to the updated sensor data.

12. The method of claim 11 , further comprising:
sending the updated computer-generated imagery to the other device via the communication link.

13. The method of claim 8 , further comprising:
identifying an object in the video;
generating an information layer including information about the object; and
overlaying the information layer onto the computer-generated imagery.

14. One or more non-transitory, computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to:
generate computer-generated imagery based on video captured from a camera;
determine a current location based on sensor data from the one or more sensors;
establish a communication link with another device;
send, to the other device via the communication link, the video, the computer-generated imagery and information indicating the current location, and
receive, from the other device, an indication of a location of the other device, wherein the computer-generated imagery indicates the location of the other device.

15. The one or more non-transitory, computer-readable storage media of claim 14 , further comprising instructions that cause the one or more processors to:
display, via a display device, the video and the computer-generated imagery, wherein the computer-generated imagery includes an indication of the current location.

16. The one or more non-transitory, computer-readable storage media of claim 14 , further comprising instructions that cause the one or more processors to:
in response to receiving user input indicating one or more annotations, generate additional computer-generated imagery based on the one or more annotations.

17. The one or more non-transitory, computer-readable storage media of claim 14 , further comprising instructions that cause the one or more processors to:
determine a route between the current location and the location of the other device; and
overlay a representation of the route on the computer-generated imagery.

18. The one or more non-transitory, computer-readable storage media of claim 14 , further comprising instructions that cause the one or more processors to:
receive an indication that motion is detected by the one or more sensors according to updated sensor data; and
update the computer-generated imagery according to the updated sensor data.

19. The one or more non-transitory, computer-readable storage media of claim 18 , further comprising instructions that cause the one or more processors to:
send the updated computer-generated imagery to the other device via the communication link.

20. The one or more non-transitory, computer-readable storage media of claim 14 , further comprising instructions that cause the one or more processors to:
identify an object in the video;
generate an information layer including information about the object; and
overlay the information layer onto the computer-generated imagery.

Assignees

• Apple Inc

Inventors

• Bilbrey Brett C
• King Nicholas V
• Pance Aleksandar

Classifications

• H04N23/632

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title

• H04N23/661

  Transmitting camera control signals through networks, e.g. control via the Internet · CPC title

• G06F1/1694 Primary

  the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer · CPC title

• H04N5/272

  Means for inserting a foreground image in a background image, i.e. inlay, outlay · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title