Method and apparatus for gesture interaction with a photo-active painted surface

Patent metadata

Field	Value
Publication number	US-9804683-B1
Application number	US-201715428751-A
Country	US
Kind code	B1
Filing date	Feb 9, 2017
Priority date	Oct 22, 2012
Publication date	Oct 31, 2017
Grant date	Oct 31, 2017

Abstract

A method and apparatus for gesture interaction with a photo-active painted surface is described. The method may include driving a spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an image to cause photo-active paint to display the image. The method may also include capturing, with at least a camera of a painted surface display system, image data of the image displayed on the photo-active paint applied to a surface and a user motion performed relative to the image. The method may also include analyzing the captured image data to determine a sequence of one or more physical movements of the user relative to the image displayed on the photo-active paint. The method may also include determining, based on the analysis, that the user motion is indicative of a gesture, and driving the spatial electromagnetic modulator to update.

First claim

We claim:

1. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to:
monitor a surface and an area in front of the surface;
determine, based on the monitoring, when a user approaches the surface or is in the area in front of the surface;
generate, based on the determination, an image of a control interface on the surface;
capture image data of the image of the control interface displayed on the surface and user motion performed relative to the image of the control interface;
analyze the captured image data to determine if the user motion is indicative of a gesture associated with the image of the control interface displayed on the surface; and
control a connected system in response to the gesture, wherein the connected system is different than a system including the processor, and wherein the control interface is associated with the connected system.

2. The non-transitory computer readable storage medium of claim 1 , wherein the connected system is one of a remote server, a home entertainment system, a home security system, or a lighting control system.

3. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:
determine, due to the monitoring, an identity of the user, wherein the image of the control interface is a specific control interface associated with the identity of the user, and wherein different control interfaces are generated for different users.

4. The non-transitory computer readable storage medium of claim 1 , wherein the gesture is one of a formal gesture, a user-created gesture, a natural gesture, or a user-interface control gesture.

5. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:
cause the image of the control interface to be updated in response to the gesture.

6. The non-transitory computer readable storage medium of claim 1 , wherein to analyze the captured image data to determine if the user motion is indicative of a gesture associated with the image of the control interface further includes instructions that, when executed by the processor cause the processor to:
analyze the captured image data to determine a sequence of one or more physical movements of the user relative to the image of the control interface, wherein the sequence of one or more physical movements is indicative of the gesture.

7. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:
determine, based on the monitoring, whether the user is approaching the surface; and
cause the image of the control interface to be enlarged in response to the determination.

8. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:
determine, based on the monitoring, a user context, wherein the user context includes an identity of the user, a user's proximity to the surface, a user's posture and attention relative to the surface, and a user's proximity to different areas of the surface.

9. The non-transitory computer readable storage medium of claim 8 , wherein the user context further determines the control interface to generate on the surface.

10. The non-transitory computer readable storage medium of claim 8 , wherein the user's proximity to different areas of the surface determines a control interface of a plurality of control interfaces to generate on the surface, and wherein different areas of the surface are associated with different control interfaces.

11. A system, comprising:
an audio-visual sensor coupled to monitor a surface and an area in front of the surface, and provide sensor data in response;
an electromagnetic (EM) modulator coupled to stimulate generation of an image on the surface, wherein photoactive compounds disposed on the surface generate the image in response to the stimulation, and wherein the EM modulator stimulates image generation in response to one or more control signals; and
a non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the system to:
receive first sensor data from the audio-visual sensor;
determine, based on the first sensor data, a context for a user with respect to the surface;
provide the one or more control signals to the EM modulator in response to the determination of the context for the user, wherein the one or more control signals cause the EM modulator to stimulate generation of an image of a control interface, the control interface associated with a connected system different than the system;
receive second sensor data from the audio-visual sensor, the second sensor data including user motion; and
determine, based on the second sensor data, that the user motion is indicative of a gesture associated with the image of the control interface displayed on the surface; and
control the connected system in response to the gesture.

12. The system of claim 11 , wherein the connected system is one of a remote server, a home entertainment system, a home security system, or a lighting control system.

13. The system of claim 11 , wherein the audio-visual sensor is a camera.

14. The system of claim 11 , wherein a photoactive paint is disposed on the surface, and wherein the photoactive paint generates the image of the control interface in response to the stimulation.

15. The system of claim 14 , wherein the photoactive paint includes one of a photoluminescent, photochromic, or thermo-chromic material.

16. The system of claim 11 , wherein the non-transitory computer readable storage medium further includes instructions that, when executed by the processor, cause the system to:
determine, based on the first sensor data, an identity of the user, wherein the image of the control interface is a specific control interface associated with the identity of the user, and wherein different control interfaces are generated for different users.

17. The system of claim 11 , wherein the gesture is one of a formal gesture, a user-created gesture, a natural gesture, or a user-interface control gesture.

18. The system of claim 11 , wherein the non-transitory computer readable storage medium further includes instructions that, when executed by the processor, cause the system to:
cause the image of the control interface to be updated in response to the gesture.

19. The system of claim 11 , wherein the user context includes an identity of the user, a user's proximity to the surface, a user's posture and attention relative to the surface, and a user's proximity to different areas of the surface.

20. The system of claim 19 , wherein the user's proximity to different areas of the surface determines a control interface of a plurality of control interfaces to generate on the surface, wherein different areas of the surface are associated with different control interfaces.

Assignees

• X Dev Llc

Inventors

• Lee Johnny
• Teller Eric
• Patrick William Graham
• Peeters Eric

Classifications

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title

• G06F3/04883 Primary

  for inputting data by handwriting, e.g. gesture or text · CPC title

• G06K9/00335

  Physics · mapped topic

• G09G5/006

  Details of the interface to the display terminal (specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04) · CPC title

• G06F3/0487

  using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser · CPC title