Systems for augmented reality sketching

Patent metadata

Field	Value
Publication number	US-11158130-B1
Application number	US-202016983798-A
Country	US
Kind code	B1
Filing date	Aug 3, 2020
Priority date	Aug 3, 2020
Publication date	Oct 26, 2021
Grant date	Oct 26, 2021

Abstract

In implementations of systems for augmented reality sketching, a computing device implements a sketch system to generate three-dimensional scene data describing a three-dimensional representation of a physical environment including a physical object. The sketch system displays a digital video in a user interface that depicts the physical environment and the physical object and the sketch system tracks movements of the physical object depicted in the digital video using two-dimensional coordinates of the user interface. These two-dimensional coordinates are projected into the three-dimensional representation of the physical environment. The sketch system receives a user input connecting a portion of a graphical element in the user interface to the physical object depicted in the digital video. The sketch system displays the portion of the graphical element as moving in the user interface corresponding to the movements of the physical object depicted in the digital video.

First claim

What is claimed is:

1. In a digital medium environment for sketching graphical elements in a user interface displaying a digital video that depicts a physical object of a physical environment, a method implemented by a computing device, the method comprising:
generating, by the computing device, three-dimensional scene data describing a three-dimensional representation of the physical environment and the physical object based on a detected surface of the physical environment;
tracking, by the computing device, movements of the physical object depicted in the digital video using two-dimensional coordinates of a center position of a determined contour for an object mask in the user interface, the object mask is based on a color of the physical object;
projecting, by the computing device, the two-dimensional coordinates of the center position of the determined contour for the object mask into the three-dimensional representation using the detected surface;
receiving, by the computing device, a user input connecting a portion of a graphical element in the user interface to the physical object depicted in the digital video; and
displaying, by the computing device, the portion of the graphical element as moving in the user interface corresponding to the movements of the physical object depicted in the digital video.

2. The method as described in claim 1 , wherein the graphical element is rendered as a line segment having a first end and a second end, the first end is connected to an additional line segment rendered in the user interface and the second end is the portion of the graphical element.

3. The method as described in claim 2 , further comprising displaying an indication of a distance between the first end and the second end of the line segment based on the movements of the physical object depicted in the digital video.

4. The method as described in claim 3 , wherein the distance corresponds to a physical distance between the physical object and a physical point in the physical environment, the physical point is determined by projecting two-dimensional coordinates of the first end of the line segment into the three-dimensional representation using the detected surface.

5. The method as described in claim 2 , further comprising displaying an indication of an angle between the line segment and the additional line segment based on the movements of the physical object depicted in the digital video.

6. The method as described in claim 2 , wherein a second additional line segment rendered in the user interface has a first end fixed to the second end of the line segment and a second end fixed to the additional line segment wherein the line segment, the additional line segment, and the second additional line segment form sides of a closed triangle shape in the user interface.

7. The method as described in claim 6 , further comprising displaying an indication of an area of the closed triangle shape based on the movements of the physical object depicted in the digital video.

8. The method as described in claim 1 , wherein the color is determined based on a color value at particular two-dimensional coordinates in the user interface defined by another user input.

9. The method as described in claim 1 , further comprising:
projecting the graphical element into the three-dimensional representation by ray casting two-dimensional coordinates of the graphical element onto the detected surface; and
generating an indication of a physical value of the physical environment based on the movements of the physical object depicted in the digital video.

10. The method as described in claim 9 , wherein the physical value is at least one of a distance, an angle, or an area.

11. The method as described in claim 1 , further comprising:
receiving an additional user input defining a variable of the graphical element as a parameter; and
displaying the variable as changing in the user interface corresponding to the movements of the physical object depicted in the digital video.

12. The method as described in claim 11 , wherein the variable of the graphical element is at least one of a size, a color, or a count.

13. The method as described in claim 11 , wherein the additional user input binds the graphical element to a virtual object rendered in the user interface and wherein the virtual object is displayed in the user interface a moving corresponding to the movements of the physical object depicted in the digital video.

14. In a digital medium environment for sketching graphical elements in a user interface displaying a digital video that depicts a physical object of a physical environment, a system comprising:
a sketch module implemented at least partially in hardware of a computing device to:
generate three-dimensional scene data describing a three-dimensional representation of the physical environment and the physical object based on a detected surface of the physical environment;
track movements of the physical object depicted in the digital video using two-dimensional coordinates of a center position of a determined contour for an object mask in the user interface, the object mask is based on a color of the physical object;
project the two-dimensional coordinates of the center position of the determined contour for the object mask into the three-dimensional representation using the detected surface;
receive a user input sketching a graphical element in the user interface, the graphical element having a portion connected to the physical object depicted in the digital video; and
display the portion of the graphical element as moving in the user interface corresponding to the movements of the physical object depicted in the digital video.

15. The system as described in claim 14 , wherein the color is determined based on a color value at particular two-dimensional coordinates in the user interface defined by another user input.

16. The system as described in claim 14 , wherein the sketch module is further implemented to:
receive an additional user input defining a connection of an additional portion of the graphical element to an additional object depicted in the digital video; and
display the additional portion of the graphical element as moving in the user interface corresponding to movements of the additional object depicted in the digital video.

17. The system as described in claim 16 , wherein the additional object is a physical object of the physical environment or a virtual object of the three-dimensional representation.

18. One or more computer-readable storage media comprising instructions stored thereon that, responsive to execution by a computing device, causes the computing device to perform operations including:
capturing a digital video depicting a physical object of a physical environment;
generating three-dimensional scene data describing a three-dimensional representation of the physical environment and the physical object based on a detected surface of the physical environment;
displaying the digital video in a user interface;
tracking movements of the physical object depicted in the digital video using two-dimensional coordinates of a center position of a determined contour for an object mask in the user interface, the object mask is based on a color of the physical object;
projecting the two-dimensional coordinates of the center position of the determined contour for the object mask into the three-dimensional representation using the detected surface;
receiving a user input connecting a first end of a line segment in the user interface to the physi

Assignees

• Adobe Inc

Inventors

• Rubaiat Habib Kazi
• Diverdi Stephen Joseph
• Suzuki Ryo
• Wei Li-Yi
• Li Wilmot Wei-Mau

Classifications

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• G06F3/04845

  for image manipulation, e.g. dragging, rotation, expansion or change of colour · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/011 Primary

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06F3/0488

  using a touch-screen or digitiser, e.g. input of commands through traced gestures · CPC title