Behind-display user interface

Patent metadata

Field	Value
Publication number	US-9939925-B2
Application number	US-201314090011-A
Country	US
Kind code	B2
Filing date	Nov 26, 2013
Priority date	Nov 26, 2013
Publication date	Apr 10, 2018
Grant date	Apr 10, 2018

Abstract

Example systems and methods of providing a user interface are presented. In one example, a graphical object is displayed on an opaque display component on a user-facing side of a computing device. Using a sensing component of the computing device, movement of a physical pointer controlled by a user is sensed. The physical pointer may be located opposite the user-facing side of the computing device. On the opaque display component, a representation of the physical pointer is displayed during the movement of the physical pointer. The graphical object, as displayed on the opaque display component, is modified based on the sensed movement of the physical pointer during the movement of the physical pointer.

First claim

What is claimed is:

1. A method of providing a user interface, the method comprising:
displaying, on an display component on a user-facing side of a computing device, a representation of a three-dimensional graphical object;
determining, using a camera of the computing device, a first apparent size and a first apparent direction of a portion of the physical pointer at a first point in time, wherein the physical pointer is controlled by a user and located opposite the user-facing side of the computing device;
determining, using the camera of the computing device, a second apparent size and a second apparent direction of the portion of the physical pointer at a second point in time;
sensing movement of the physical pointer based at least in part on a difference in magnitude between the first apparent size and the second apparent size of the portion of the physical pointer and a difference between the first apparent direction and the second apparent direction of the portion of the physical pointer, wherein the movement includes at least a variance in an orientation of the physical pointer;
displaying, on the opaque display component, a representation of the physical pointer, captured by the camera, during the movement of the physical pointer; and
updating, as displayed on the display component, the representation of the three-dimensional graphical object, based on the sensed movement of the physical pointer during the movement of the physical pointer, wherein updating the representation of the three-dimensional graphical object includes at least a modification to a shape of the representation of the three-dimensional graphical object that includes modifying a distance between a first corner and a second corner of the three-dimensional graphical object such that the modified shape is stretched relative to the unmodified shape of the representation of the three-dimension graphical object and a type of the modification is based on the variance in the orientation of the physical pointer.

2. The method of claim 1 , wherein the physical pointer comprises a hand-held device being controlled by the user.

3. The method of claim 1 , wherein the physical pointer comprises a portion of a body of the user.

4. The method of claim 1 , wherein the camera is a first camera, the method further comprising:
sensing, using a second camera of the computing device, a location of the user relative to the user-facing side of the computing device; and
adjusting the representation of the three-dimensional graphical object, as displayed on the display component, based on the sensed location of the user relative to the user-facing side of the computing device.

5. The method of claim 1 , further comprising:
determining, using the camera of the computing device, a first apparent shape of the portion of the physical pointer at the first point in time; and
determining, using the camera of the computing device, a second apparent shape of the portion of the physical pointer at the second point in time, and
wherein sensing movement of the physical pointer is based further on a change in shape of the physical pointer between the first apparent shape and the second apparent shape.

6. The method of claim 1 , wherein sensing the movement of the physical pointer is based further on a plurality of marks visible on a background surface that is positioned opposite the physical pointer from the computing device.

7. The method of claim 6 , further comprising:
detecting a change in location or a change in orientation of the background surface relative to the computing device; and
altering, as displayed on the display component, at least one of an apparent location of the representation of the three-dimensional graphical object or an apparent orientation of the representation of the three-dimensional graphical object based on the detected change in location or change in orientation of the background surface relative to the computing device.

8. The method of claim 7 , wherein detecting the change in location or change in orientation of the background surface is based on detecting movement of the plurality of marks visible on the background surface.

9. The method of claim 1 , further comprising:
displaying, on the display component, a virtual shadow cast by the representation of the three-dimensional graphical object onto a representation of a physical background surface captured by the camera.

10. A non-transitory computer-readable storage medium comprising instructions that, when executed by at least one processor of a computing device, cause the computing device to perform operations comprising:
displaying, on an display component on a user-facing side of the computing device, a representation of a graphical object;
determining, using a camera of the computing device, a first apparent size and a first apparent shape of a portion of a physical pointer at a first point in time, wherein the physical pointer is controlled by a user and located opposite the user-facing side of the computing device;
determining, using the camera of the computing device, a second apparent size, and a second apparent shape of the portion of the physical pointer at a second point in time;
sensing movement of the physical pointer based at least in part on a difference in magnitude between the first apparent size and the second apparent size of the portion of the physical pointer, difference between the first apparent direction and the second apparent direction of the of the physical pointer, and a difference between the first apparent shape and the second apparent shape of the portion of the physical pointer, wherein the movement includes at least a variance in orientation of the physical pointer;
displaying, on the display component, a representation of the physical pointer, captured by the camera, during the movement of the physical pointer; and
updating, as displayed on the display component, the representation of the graphical object based on the sensed movement of the physical pointer during the movement of the physical pointer, wherein updating the representation of the graphical object includes at least a modification to shape of the representation of the graphical object that includes modifying a distance between a first corner and a second corner of the three-dimensional graphical object such that the modified shape is stretched relative to the unmodified shape of the representation of the three-dimension graphical object and a type of the modification is based on the variance in the orientation of the physical pointer.

11. The storage medium of claim 10 , wherein the physical pointer comprises a hand-held device being controlled by the user.

12. The storage medium of claim 10 , wherein the physical pointer comprises a portion of a body of the user.

13. The storage medium of claim 10 , wherein the camera is a first camera, the operations further comprising:
sensing, using a second camera of the comprising device, a location of the user relative to the user-facing side of the computing device; and
adjusting the representation of the three-dimensional graphical object, as displayed on the opaque display component, based on the sensed location of the user relative to the user-facing side of the computing device.

14. A computing device comprising:
an display component on a user-facing side of the computing device;
a camera;
at least one processor; and
memory having instructions stored thereon that, when executed by the at least one processor, cause the computing device to:
display, on the display component, a representation of a three-dimensional graphical object;
determine, us

Assignees

• Adobe Systems Inc

Inventors

• Ito Daichi
• Carr Nathan
• Sasaki Naoki

Classifications

• G06F3/0346 Primary

  with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors · CPC title

• G06F3/03545

  Pens or stylus · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title

• G06F3/005

  Input arrangements through a video camera · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title