Systems for Augmented Reality Sketching

Patent metadata

Field	Value
Publication number	US-2022148267-A1
Application number	US-202117511385-A
Country	US
Kind code	A1
Filing date	Oct 26, 2021
Priority date	Aug 3, 2020
Publication date	May 12, 2022
Grant date	—

Abstract

In implementations of systems for augmented reality sketching, a computing device implements a sketch system to generate three-dimensional scene data describing a three-dimensional representation of a physical environment including a physical object. The sketch system displays a digital video in a user interface that depicts the physical environment and the physical object and the sketch system tracks movements of the physical object depicted in the digital video using two-dimensional coordinates of the user interface. These two-dimensional coordinates are projected into the three-dimensional representation of the physical environment. The sketch system receives a user input connecting a portion of a graphical element in the user interface to the physical object depicted in the digital video. The sketch system displays the portion of the graphical element as moving in the user interface corresponding to the movements of the physical object depicted in the digital video.

First claim

What is claimed is:

1 . In a digital medium environment, a method implemented by a computing device, the method comprising:
receiving, by the computing device, three-dimensional scene data describing a three-dimensional representation of a physical environment and a physical object; displaying, by the computing device, a digital video in a user interface depicting the physical environment and the physical object; receiving, by the computing device, a user input specifying two-dimensional coordinates in the user interface corresponding to the physical object; and tracking, by the computing device, movements of the physical object in the physical environment and in the digital video by projecting the two-dimensional coordinates into the three-dimensional representation.

2 . The method as described in claim 1 , further comprising:
receiving an additional user input sketching a graphical element in the user interface; and displaying the graphical element as moving in the user interface corresponding to the movements of the physical object in the physical environment.

3 . The method as described in claim 2 , further comprising displaying a variable of the graphical element as changing in the user interface corresponding to the movements of the physical object in the physical environment.

4 . The method described in claim 3 , wherein the variable is a count, a color, or a size.

5 . The method as described in claim 1 , wherein projecting the two-dimensional coordinates into the three-dimensional representation is based on a detected surface of the physical environment.

6 . The method as described in claim 5 , further comprising ray casting the two-dimensional coordinates onto the detected surface.

7 . The method as described in claim 1 , wherein tracking movements of the physical object in the digital video is based at least partially on a color of the physical object.

8 . The method as described in claim 1 , wherein tracking movements of the physical object in the digital video is based at least partially on an object mask.

9 . The method as described in claim 1 , further comprising generating an indication of a physical value of the physical environment based on the movements of the physical object in the digital video.

10 . The method as described in claim 9 , wherein the physical value is a distance, an angle, or an area.

11 . A system comprising:
one or more processors; and a computer-readable storage medium storing instructions that are executable by the one or more processors to perform operations comprising:
receiving three-dimensional scene data describing a three-dimensional representation of a physical environment and a physical object;
displaying a digital video in a user interface depicting the physical environment and the physical object;
receiving a user input specifying a color of the physical object; and
tracking movements of the physical object in the physical environment and in the digital video by projecting two-dimensional coordinates corresponding to the color into the three-dimensional representation.

12 . The system as described in claim 11 , wherein projecting the two-dimensional coordinates into the three-dimensional representation is based on a detected surface of the physical environment.

13 . The system as described in claim 11 , wherein tracking movements of the physical object in the digital video is based at least partially on a color of the physical object.

14 . The system as described in claim 11 , wherein tracking movements of the physical object in the digital video is based at least partially on an object mask.

15 . The system as described in claim 11 , wherein the operations further comprise:
receiving an additional user input sketching a graphical element in the user interface; and displaying the graphical element as moving in the user interface corresponding to the movements of the physical object in the physical environment.

16 . A system comprising:
means for receiving three-dimensional scene data describing a three-dimensional representation of a physical environment and a physical object; means for receiving video data describing a digital video depicting the physical environment and the physical object; means for receiving a user input specifying two-dimensional coordinates in a user interface corresponding to the physical object depicted in the digital video; and means for displaying a graphical element as moving in the user interface corresponding to movements of the physical object in the physical environment.

17 . The system as described in claim 16 , wherein projecting the two-dimensional coordinates into the three-dimensional representation is based on a detected surface of the physical environment.

18 . The system as described in claim 16 , wherein tracking movements of the physical object in the digital video is based at least partially on a color of the physical object.

19 . The system as described in claim 16 , wherein tracking movements of the physical object in the digital video is based at least partially on an object mask.

20 . The system as described in claim 16 , further comprising means for generating an indication of a physical value of the physical environment based on the movements of the physical object in the digital video.

Assignees

• Adobe Inc

Inventors

• Rubaiat Habib Kazi
• Diverdi Stephen Joseph
• Suzuki Ryo
• Wei Li-Yi
• Li Wilmot Wei-Mau

Classifications

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title