Synchronized, interactive augmented reality displays for multifunction devices

Patent metadata

Field	Value
Publication number	US-11721073-B2
Application number	US-202017104305-A
Country	US
Kind code	B2
Filing date	Nov 25, 2020
Priority date	Jan 5, 2010
Publication date	Aug 8, 2023
Grant date	Aug 8, 2023

Abstract

A device can receive live video of a real-world, physical environment on a touch sensitive surface. One or more objects can be identified in the live video. An information layer can be generated related to the objects. In some implementations, the information layer can include annotations made by a user through the touch sensitive surface. The information layer and live video can be combined in a display of the device. Data can be received from one or more onboard sensors indicating that the device is in motion. The sensor data can be used to synchronize the live video and the information layer as the perspective of video camera view changes due to the motion. The live video and information layer can be shared with other devices over a communication link.

First claim

What is claimed is:

1. A system, comprising:
a camera;
one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:
generate computer-generated imagery based on video captured from the camera;
establish a communication link with another device;
synchronize display of the video with the other device via the communication link; and
during the synchronized display of the video:
send, to the other device via the communication link, the computer-generated imagery generated based on the video; and
in response to receiving, from the other device, an indication of one or more annotations applied to the video by the other device, update the computer-generated imagery based at least on the one or more annotations from the other device.

2. The system of claim 1 , wherein the memory further comprises additional instructions that further cause the one or more processors to:
in response to receiving user input indicating one or more additional annotations applied to the video, generate additional computer-generated imagery based on the one or more additional annotations applied to the video.

3. The system of claim 2 , wherein the memory further comprises additional instructions that further cause the one or more processors to:
send, to the other device via the communication link, an indication of the one or more additional annotations to the video.

4. The system of claim 1 , further comprising:
one or more sensors;
wherein the memory further comprises additional instructions that further cause the one or more processors to:
determine a current location based on sensor data from the one or more sensors, wherein the computer-generated imagery includes an indication of the current location.

5. The system of claim 1 , further comprising:
a display configured to display the video and the computer-generated imagery.

6. The system of claim 1 , wherein the memory further comprises additional instructions that further cause the one or more processors to:
identify an object in the video;
generate an information layer including information about the object; and
overlay the information layer onto the computer-generated imagery.

7. The system of claim 6 , further comprising:
one or more sensors;
wherein the memory further comprises additional instructions that further cause the one or more processors to:
determine a current orientation of the camera relative to the object based on sensor data from the one or more sensors; and
send, to the other device via the communication link, information indicating the current orientation.

8. A method, comprising:
generating computer-generated imagery based on video captured from a camera of a device;
establishing a communication link with another device;
synchronizing display of the video with the other device via the communication link; and
during the synchronized display of the video:
sending, to the other device via the communication link, the computer-generated imagery generated based on the video;
receiving, from the other device, an indication of one or more annotations applied to the video by the other device; and
updating the computer-generated imagery based at least on the one or more annotations to the video from the other device.

9. The method of claim 8 , further comprising:
receiving, at the device, user input indicating one or more additional annotations; and
generating additional computer-generated imagery based on the one or more additional annotations applied to the video.

10. The method of claim 8 , further comprising:
sending, to the other device via the communication link, an indication of the one or more additional annotations applied to the video.

11. The method of claim 8 , further comprising:
determining a current location based on sensor data from one or more sensors of the device, wherein the computer-generated imagery includes an indication of the current location.

12. The method of claim 8 , further comprising:
displaying the video and the computer-generated imagery via a display of the device.

13. The method of claim 8 , further comprising:
identifying an object in the video;
generating an information layer including information about the object; and
overlaying the information layer onto the computer-generated imagery.

14. The method of claim 13 , further comprising:
determine a current orientation of the camera relative to the object based on sensor data from one or more sensors of the device; and
sending, to the other device via the communication link, information indicating the current orientation.

15. One or more non-transitory, computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to:
generate computer-generated imagery based on video captured from a camera;
establish a communication link with another device;
synchronize display of the video with the other device via the communication link; and
during the synchronized display of the video:
send, to the other device via the communication link, the computer-generated imagery generated based on the video; and
in response to receiving, from the other device, an indication of one or more annotations applied to the video by the other device, update the computer-generated imagery based at least on the one or more annotations to the video from the other device.

16. The one or more non-transitory, computer-readable storage media of claim 15 , further comprising additional instructions that cause the one or more processors to:
in response to receiving user input indicating one or more additional annotations applied to the video, generate additional computer-generated imagery based on the one or more additional annotations.

17. The one or more non-transitory, computer-readable storage media of claim 15 , further comprising additional instructions that cause the one or more processors to:
send, to the other device via the communication link, an indication of the one or more additional annotations applied to the video.

18. The one or more non-transitory, computer-readable storage media of claim 15 , further comprising additional instructions that cause the one or more processors to:
determine a current location based on sensor data from one or more sensors, wherein the computer-generated imagery includes an indication of the current location.

19. The one or more non-transitory, computer-readable storage media of claim 15 , further comprising additional instructions that cause the one or more processors to:
identify an object in the video;
generate an information layer including information about the object; and
overlay the information layer onto the computer-generated imagery.

20. The one or more non-transitory, computer-readable storage media of claim 19 , further comprising additional instructions that cause the one or more processors to:
determine a current orientation of the camera relative to the object based on sensor data from one or more sensors; and
send, to the other device via the communication link, information indicating the current orientation.

Assignees

• Apple Inc

Inventors

• Bilbrey Brett C
• King Nicholas V
• Pance Aleksandar

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F1/1694 Primary

  the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G06T19/20

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title