Remote control of a desktop application via a mobile device

Patent metadata

Field	Value
Publication number	US-10048762-B2
Application number	US-201314072560-A
Country	US
Kind code	B2
Filing date	Nov 5, 2013
Priority date	Nov 5, 2013
Publication date	Aug 14, 2018
Grant date	Aug 14, 2018

Abstract

One embodiment of the present invention provides a system for using a mobile device to remotely control a desktop application that was configured for use with a pointing device. During operation, the system receives at a mobile device, from a user, a connection request to connect to a desktop application executing on a remote device. In response to the connection request, the system establishes a remote control session between the mobile device and the remote device. Next, the system receives, at the mobile device, a video stream from a first camera coupled to the mobile device. The system then analyzes the video stream, at the mobile device, to identify a gesture made by the user. Next, the system identifies, at the mobile device, a pointing device command corresponding to the gesture. Finally, the system sends the pointing device command to the desktop application executing on the remote device.

First claim

What is claimed is:

1. A computer-implemented method for using a mobile device to remotely control a desktop application that is configured for use with a pointing device, the method comprising:
receiving, at a mobile device from a user, a connection request to connect to a desktop application executing on a remote device;
establishing, at the mobile device, a remote control session between the mobile device and the remote device in response to the connection request;
receiving, at the mobile device, a first video stream from a first camera coupled to the mobile device, wherein the first camera is coupled to an opposite side of the mobile device from a touch screen display;
receiving, via the touch screen display, input from the user selecting an area of the touch screen display corresponding to an image of a hand of the user, wherein the image of the hand of the user is a portion of the first video stream from the first camera;
engaging, at the mobile device, a focus lock feature based on the input from the user, wherein the focus lock feature is configured to aid in analyzing gestures by setting a focus of the first camera based on the location of the hand of the user;
analyzing the first video stream, at the mobile device using hardware within the mobile device that is configured for determining motion vectors, to identify a hand gesture made by the user, wherein the hand gesture is detected within the first video stream from the first camera;
identifying, at the mobile device, a pointing device command corresponding to the hand gesture; and
sending the pointing device command from the mobile device to the desktop application executing on the remote device,
wherein the pointing device command is configured to cause a cursor to move within the desktop application executing on the remote device.

2. The computer-implemented method of claim 1 , further comprising:
receiving, at the mobile device, a second video stream from a second camera coupled to the mobile device;
analyzing the second video stream, at the mobile device, to identify a second gesture made by the user, wherein the second gesture is detected within the second video stream from the second camera;
identifying, at the mobile device, a second pointing device command corresponding to the second gesture; and
sending the second pointing device command from the mobile device to the desktop application executing on the remote device.

3. The computer-implemented method of claim 2 , wherein the second camera is coupled to a same side of the mobile device as the touch screen display.

4. The computer-implemented method of claim 2 , wherein identifying the second gesture involves identifying a motion of one of an eye or an eyelid.

5. The computer-implemented method of claim 1 , further comprising using an accelerometer coupled to the mobile device to compensate for a motion of the mobile device while the first video stream is being received.

6. The computer-implemented method of claim 1 , wherein the mobile device is a wearable device.

7. A non-transitory computer-readable storage medium storing instructions that when executed by a mobile device cause the mobile device to perform a method for remotely controlling a desktop application that is configured for use with a pointing device, the method comprising:
receiving, at a mobile device from a user, a connection request to connect to a desktop application executing on a remote device;
establishing, at the mobile device, a remote control session between the mobile device and the remote device in response to the connection request;
receiving, at the mobile device, a first video stream from a first camera coupled to the mobile device, wherein the first camera is coupled to an opposite side of the mobile device from a touch screen display;
receiving, via the touch screen display input from the user selecting an area of the touch screen display corresponding to an image of a hand of the user, wherein the image of the hand of the user is a portion of the first video stream from the first camera;
engaging, at the mobile device, a focus lock feature based on the input from the user, wherein the focus lock feature is configured to aid in analyzing gestures by setting a focus of the first camera based on the location of the hand of the user;
analyzing the first video stream, at the mobile device using hardware within the mobile device that is configured for determining motion vectors, to identify a hand gesture made by the user, wherein the hand gesture is detected within the first video stream from the first camera;
identifying, at the mobile device, a pointing device command corresponding to the hand gesture; and
sending the pointing device command from the mobile device to the desktop application executing on the remote device,
wherein the pointing device command is configured to cause a cursor to move within the desktop application executing on the remote device.

8. The non-transitory computer-readable storage medium of claim 7 , wherein the method further comprises:
receiving, at the mobile device, a second video stream from a second camera coupled to the mobile device;
analyzing the second video stream, at the mobile device, to identify a second gesture made by the user, wherein the second gesture is detected within the second video stream from the second camera;
identifying, at the mobile device, a second pointing device command corresponding to the second gesture; and
sending the second pointing device command from the mobile device to the desktop application executing on the remote device.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the second camera is coupled to a same side of the mobile device as the touch screen display.

10. The non-transitory computer-readable storage medium of claim 8 , wherein identifying the second gesture involves identifying a motion of one of an eye or an eyelid.

11. The non-transitory computer-readable storage medium of claim 7 , wherein the method further comprises using an accelerometer coupled to the mobile device to compensate for a motion of the mobile device while the first video stream is being received.

12. The non-transitory computer-readable storage medium of claim 7 , wherein the mobile device is a wearable device.

13. A mobile device configured to remotely control a desktop application that is configured for use with a pointing device, comprising:
a memory;
a processor; and
a storage medium storing instructions that, when executed by the processor, cause the mobile device to perform a method, comprising:
receiving, at a mobile device from a user, a connection request to connect to a desktop application executing on a remote device;
establishing, at the mobile device, a remote control session between the mobile device and the remote device in response to the connection request;
receiving, at the mobile device, a first video stream from a first camera coupled to the mobile device, wherein the first camera is coupled to an opposite side of the mobile device from a touch screen display;
receiving, via the touch screen display, input from the user selecting an area of the touch screen display corresponding to an image of a hand of the user, wherein the image of the hand of the user is a portion of the first video stream from the first camera;
engaging, at the mobile device, a focus lock feature based on the input from the user, wherein the focus lock feature is configured to aid in analyzing gestures by setting a focus of the first camera based on the location of the hand of the user;
analyzing the first video stream, at the mobile device using hardwa

Assignees

• Intuit Inc

Inventors

• Lesner Christopher Z

Classifications

• G06F3/048

  Interaction techniques based on graphical user interfaces [GUI] · CPC title

• H04L67/025

  for remote control or remote monitoring of applications · CPC title

• G06F3/0304

  Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042) · CPC title

• G06F3/013

  Eye tracking input arrangements (G06F3/015 takes precedence) · CPC title

• G06F3/017 Primary

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title