Shared augmented reality session recording and playback

Patent metadata

Field	Value
Publication number	US-12079439-B2
Application number	US-202217889364-A
Country	US
Kind code	B2
Filing date	Aug 16, 2022
Priority date	May 30, 2017
Publication date	Sep 3, 2024
Grant date	Sep 3, 2024

Abstract

Augmented reality (AR) or virtual reality (VR) systems described herein can be configured to record images, video, and/or annotations for concurrent communication to a remote system for display or subsequent access. A communication between a user and an expert user using the system can include an audio communication (unidirectional or bidirectional), a video communication from the user to the expert user (allowing the expert user to see, in real time, the same environment as the user), and a data communication (via which content overlaid over the video communication, such as annotations, may be displayed for both the expert user and the user). The systems can be configured to communicate concurrently while also creating a recording for later review based on a “live” or current issue being faced by a user assisted by an expert where the communication session is recorded for later playback by other users.

First claim

The invention claimed is:

1. A method of person-to-person communication session playback, the method comprising:
initiating a person-to-person communication session between a first device operated by a first user and a second device operated by a second user;
capturing a live video stream with a camera of the first device during the communication session;
sending at least a portion of the live video stream from the first device to the second device;
receiving with the first device at least one audio-visual communication specified by the second user during the communication session;
providing the audio-visual communication to the first user via an output device of the first device;
storing the audio-visual communication for playback to a third user of a third device outside of a person-to-person communication session;
detecting a period of time during the communication session in which the first device does not receive any audio-visual communication specified by the second user; and
marking the period of time such that the period of time is not included in playback of the communication session.

2. The method of claim 1 , wherein the audio-visual communication is stored independent of a viewing position of the first device.

3. The method of claim 1 , further comprising:
generating a three-dimensional (3D) reconstruction of a first environment of the first device from the live video stream during the communication session; and
storing the 3D reconstruction of the first environment for the playback to the third user.

4. The method of claim 3 , wherein the 3D reconstruction of the first environment comprises a plurality of feature points derived from the live video stream, the feature points enabling the tracking of a second environment during playback of the audio-visual communication with the third device.

5. The method of claim 1 , further comprising:
detecting a first object within the live video stream; and
anchoring the audio-visual communication to the detected first object,
wherein the storing of the audio-visual communication comprises storing an indication of the anchoring of the audio-visual communication to the detected first object independent of the live video stream.

6. The method of claim 5 , further comprising:
initiating a playback session for the third user outside of the person-to-person communication session;
capturing an image with a camera of the third device;
detecting a second object with the captured image, the second object being the same as or similar to the first object; and
displaying the audio-visual communication anchored to the detected second object.

7. The method of claim 1 , wherein the audio-visual communication comprises at least one of: a verbal instructions and a graphical annotation.

8. The method of claim 1 , further comprising initiating a playback session for the third user in a virtual environment outside of the person-to-person communication session.

9. The method of claim 1 , wherein the person-to-person communication session comprises an augmented reality video session.

10. A first device for person-to-person communication session playback, the device comprising:
a camera;
an output device;
a processor; and
a memory in communication with the processor and having stored thereon computer-executable instructions to cause the processor to:
initiate a person-to-person communication session between the first device operated by a first user and a second device operated by a second user;
capture a live video stream with the camera during the communication session;
send at least a portion of the live video stream to the second device;
receive at least one audio-visual communication specified by the second user during the communication session;
provide the audio-visual communication to the first user via the output device;
store the audio-visual communication in the memory for playback to a third user of a third device outside of a person-to-person communication session;
detect a period of time during the communication session in which the first device does not receive any audio-visual communication specified by the second user; and
mark the period of time such that the period of time is not included in playback of the communication session.

11. The first device of claim 10 , wherein the audio-visual communication is stored independent of a viewing position of the first device.

12. The first device of claim 10 , wherein the memory further has stored thereon computer-executable instructions to cause the processor to:
generate a three-dimensional (3D) reconstruction of a first environment of the first device from the live video stream during the communication session; and
store the 3D reconstruction of the first environment in the memory for the playback to the third user.

13. The first device of claim 12 , wherein the 3D reconstruction of the first environment comprises a plurality of feature points derived from the live video stream, the feature points enabling the tracking of a second environment during playback of the audio-visual communication with the third device.

14. The first device of claim 10 , wherein the memory further has stored thereon computer-executable instructions to cause the processor to:
detect a first object within the live video stream; and
anchor the audio-visual communication to the detected first object,
wherein the storing of the audio-visual communication comprises storing an indication of the anchoring of the audio-visual communication to the detected first object independent of the live video stream.

15. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a processor, cause a first device to:
initiate a person-to-person communication session between the first device operated by a first user and a second device operated by a second user;
capture a live video stream with a camera of the first device during the communication session;
send at least a portion of the live video stream from the first device to the second device;
receive at least one audio-visual communication specified by the second user during the communication session;
provide the audio-visual communication to the first user via an output device of the first device;
store the audio-visual communication in the memory for playback to a third user of a third device outside of a person-to-person communication session;
detect a period of time during the communication session in which the first device does not receive any audio-visual communication specified by the second user; and
mark the period of time such that the period of time is not included in playback of the communication session.

16. The non-transitory computer readable storage medium of claim 15 , wherein the audio-visual communication is stored independent of a viewing position of the first device.

17. The non-transitory computer readable storage medium of claim 15 , further having stored thereon instructions that, when executed by the processor, cause the first device to:
generate a three-dimensional (3D) reconstruction of a first environment of the first device from the live video stream during the communication session; and
store the 3D reconstruction of the first environment in the memory for the playback to the third user.

18. The non-transitory computer readable storage medium of claim 17 , wherein the 3D reconstruction of the first environment comprises a plurality of feature points derived from the live video stream, the feature points enabling the

Assignees

• Ptc Inc

Inventors

• Wright Jr Gerald
• Lechleiter John James
• Gervautz Michael
• Gauglitz Steffen
• Kolsch Mathias
• Mahendran Arungundram

Classifications

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06Q10/20

  Administration of product repair or maintenance · CPC title

• G06F3/167

  Audio in a user interface, e.g. using voice commands for navigating, audio feedback · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title