Generating augmented reality objects on real-world surfaces using a digital writing device

Patent metadata

Field	Value
Publication number	US-10825253-B2
Application number	US-201916375549-A
Country	US
Kind code	B2
Filing date	Apr 4, 2019
Priority date	Sep 26, 2017
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

The present disclosure includes systems, methods, computer readable media, and devices that can generate accurate augmented reality objects based on tracking a writing device in relation to a real-world surface. In particular, the systems and methods described herein can detect an initial location of a writing device, and further track movement of the writing device on a real-world surface based on one or more sensory inputs. For example, disclosed systems and methods can generate an augmented reality object based on pressure detected at a tip of a writing device, based on orientation of the writing device, based on motion detector elements of the writing device (e.g., reflective materials, emitters, or object tracking shapes), and/or optical sensors. The systems and methods further render augmented reality objects within an augmented reality environment that appear on the real-world surface based on tracking the movement of the writing device.

First claim

What is claimed is:

1. In a digital medium environment for generating augmented reality objects using a digital writing device, a computer-implemented method for creating a digital object in an augmented reality environment corresponding to a real-world surface, the computer-implemented method comprising:
determining an initial location of a writing device on a real-world surface;
detecting real-world movement of the writing device relative to the initial location by:
determining that the real-world movement of the writing device is above a speed threshold;
in response to determining that the real-world movement is above the speed threshold, determining the real-world movement of the writing device based on a first combination of two or more tracking techniques; and
in response to determining that the real-world movement is at or below the speed threshold, determining the real-world movement of the writing device based on a second combination of two or more tracking techniques; and

rendering, within an augmented reality environment generated by way of an augmented reality device, an augmented reality object that appears to be drawn on the real-world surface based on the real-world movement of the writing device.

2. The computer-implemented method of claim 1 , wherein the two or more tracking techniques comprise two or more of:
tracking real-world movement of a reflective element located on the writing device;
utilizing an emitter associated with the writing device to track real-world movement;
tracking real-world movement utilizing an optical sensor located on the writing device;
utilizing an inertial measurement unit to track real-world movement of the writing device;
utilizing an object recognition technique to track real-world movement of the writing device; or
implementing a machine learning model to track real-world movement of the writing device.

3. The computer-implemented method of claim 1 , further comprising:
analyzing, by way of the augmented reality device, a plurality of real-world surfaces within a real-world environment to identify, from the plurality of real-world surfaces, the real-world surface as sufficiently large and sufficiently flat for composing; and
providing, within the augmented reality environment and in response to identifying the real-world surface as sufficiently large and sufficiently flat for composing, a visual indication that the real-world surface is acceptable for composing.

4. The computer-implemented method of claim 3 , further comprising:
identifying, from the plurality of real-world surfaces, one or more real-world surfaces that are unacceptable for composing; and
providing, within the augmented reality environment and in response to identifying the one or more real-world surfaces as unacceptable for composing, a visual indication that the one or more real-world surfaces are unacceptable for composing.

5. The computer-implemented method of claim 1 , further comprising:
determining environmental factors associated with tracking the writing device; and
utilizing the two or more tracking techniques based on the environmental factors.

6. The computer-implemented method of claim 5 , wherein determining the environmental factors comprises one or more of determining a reflectiveness of the real-world surface or detecting lighting conditions associated with the augmented reality environment.

7. The computer-implemented method of claim 1 , wherein rendering the augmented reality object comprises:
detecting, by way of a pressure sensor associated with the writing device, that the writing device is contacting the real-world surface; and
generating, within the augmented reality environment and in response to detecting that the writing device is contacting the real-world surface, a digital mark that appears to be drawn on the real-world surface and that follows the real-world movement of the writing device.

8. The computer-implemented method of claim 7 , further comprising:
detecting, by way of the pressure sensor associated with the writing device, a pressure force applied to the writing device;
detecting, by way of an inertial measurement unit associated with the writing device, a tilt of the writing device; and
adjusting a compositional attribute of the digital mark based on the pressure force applied to the writing device and the tilt of the writing device.

9. A system comprising:
at least one processor; and
a non-transitory computer readable medium comprising instructions that, when executed by the at least one processor, cause the system to:
determine an initial location of a writing device on a real-world surface by detecting a locator element associated with the writing device in response to receiving an indication of an initial contact of the writing device on the real-world surface;
detect real-world movement of the writing device relative to the initial location by:
determining that the real-world movement of the writing device is above a speed threshold;
in response to determining that the real-world movement is above the speed threshold, determining the real-world movement of the writing device based on a first combination of two or more tracking techniques; and
in response to determining that the real-world movement is at or below the speed threshold, determining the real-world movement of the writing device based on a second combination of two or more tracking techniques; and

render, within an augmented reality environment generated by way of an augmented reality device, an augmented reality object that appears to be drawn on the real-world surface based on the real-world movement of the writing device.

10. The system of claim 9 , wherein the two or more tracking techniques comprise two or more of:
tracking real-world movement of a reflective element located on the writing device;
utilizing an emitter associated with the writing device to track real-world movement;
tracking real-world movement utilizing an optical sensor located on the writing device;
utilizing an inertial measurement unit to track real-world movement of the writing device;
utilizing an object recognition technique to track real-world movement of the writing device; or
implementing a machine learning model to track real-world movement of the writing device.

11. The system of claim 9 , further comprising instructions that, when executed by the at least one processor, cause the system to analyze, by way of the augmented reality device, a real-world environment to identify at least one real-world surface that is sufficiently large and sufficiently flat for composing.

12. The system of claim 11 , further comprising instructions that, when executed by the at least one processor, cause the system to provide, within the augmented reality environment and in response to analyzing the at least one real-world surface, an indication that the at least one real-world surface is acceptable for composing.

13. The system of claim 12 , further comprising instructions that, when executed by the at least one processor, cause the system to:
identify, from a plurality of real-world surfaces, one or more real-world surfaces that are unacceptable for composing; and
provide, within the augmented reality environment and in response to identifying the one or more real-world surfaces as unacceptable for composing, a visual indication that the one or more real-world surfaces are unacceptable for composing.

14. The system of claim 9 , further comprising instructions that, when executed by the at least one process, cause the system to:
determine environmental factors associated with tracking the wri

Assignees

• Adobe Inc

Inventors

• Rhodes Tenell
• Miller Gavin S P
• Aksit Duygu Ceylan
• Ito Daichi

Classifications

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/03545

  Pens or stylus · CPC title

• G06F3/0346

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title