Generating summary of activity on computer gui

Patent metadata

Field	Value
Publication number	US-2016349928-A1
Application number	US-201514722215-A
Country	US
Kind code	A1
Filing date	May 27, 2015
Priority date	May 27, 2015
Publication date	Dec 1, 2016
Grant date	—

Abstract

Auditing activity of a user on a user computing device is provided. When the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), an active window is identified within a captured screenshot by locating a screen location of the HID input in the GUI. The active window along with an extracted textual title of the active window, the screen location of the HID input and a user identifier are stored to a data store.

First claim

What is claimed is:

1 . A method for auditing activity of a user on a user computing device, the method comprising:
receiving, by a computer, a screenshot of a display device of the user computing device when the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), and a screen location of the HID input, and a user identifier; identifying, by the computer, an active window in the received screenshot, based on the screen location of the HID input; extracting, by the computer, from the screenshot, the identified active window; determining, by the computer, a textual title of the active window; and storing one or more electronic records, by the computer, comprising the extracted active window, the textual title of the active window, and the user identifier to a data store.

2 . A method according to claim 1 , wherein identifying the active window comprises:
locating edges of one or more rectangles present in the received screenshot; identifying one or more rectangles corresponding to the located edges; detecting one or more windows corresponding to the identified rectangles; and identifying the active window based on the screen location of the HID input within the detected one or more windows.

3 . A method according to claim 1 , wherein determining the textual title of the active window comprises:
recognizing characters in a general region of a title bar of the active window using an object character recognition (OCR) technique; and extracting the characters to develop the textual title of the active window.

4 . A method according to claim 1 , wherein the HID includes one or more of: a computer mouse, a keyboard, a touchscreen, and an audio input.

5 . A method according to claim 1 ,
wherein receiving a screenshot of a display device further comprises receiving one or more of: a device identifier, a network identifier, an active user program identifier, raw HID inputs, and a date/time stamp; and wherein the storing one or more electronic records further comprises storing one or more of: the device identifier, the network identifier, the active user program identifier, the raw HID inputs, and the date/time stamp.

6 . A method according to claim 1 , furthering comprising:
replacing the textual title of the active window with an internal dictionary key.

7 . A method according to claim 1 , wherein storing the one or more electronic records further comprises:
indexing the textual title of the active window to enable querying by the computer in a GUI environment.

8 . A method according to claim 1 , further comprising:
retrieving the extracted active window, the textual title of the active window and the user identifier from the data store; and displaying the extracted active window, the textual title of the active window and the user identifier on a display device of the computer.

9 . A method according to claim 1 , wherein steps of the method are performed using a set of instructions of a program, wherein the program is an application executed on one or more of:
a cloud computing environment; a mobile device; and a desktop device.

10 . A method according to claim 1 , further comprising:
loading, by the computer, the one or more electronic records; and displaying, by the computer on a display device of the computer, the loaded one or more electronic records including one or more of: the extracted active window; the textual title of the active window; and the user identifier.

11 . A method according to claim 1 , further comprising:
searching, by the computer, an index of the one or more electronic records based on a query; ranking results of the searching; and displaying the results according to the ranking.

12 . A computer program product for auditing activity of a user on a user computing device, the computer program product comprising:
one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the program instructions comprising: program instructions to receive, by a computer, a screenshot of a display device of the user computing device when the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), and a screen location of the HID input, and a user identifier; program instructions to identify, by the computer, an active window in the received screenshot, based on the screen location of the HID input; program instructions to extract, by the computer, from the screenshot, the identified active window; program instructions to determine, by the computer, a textual title of the active window; and program instructions to store one or more electronic records, by the computer, comprising the extracted active window, the textual title of the active window, and the user identifier to a data store.

13 . A computer system for auditing activity of a user on a user computing device, the computer system comprising:
one or more computer processors; one or more computer-readable storage media; program instructions stored on the computer-readable storage media for execution by at least one of the one or more processors, the program instructions comprising: program instructions to receive, by a computer, a screenshot of a display device of the user computing device when the user computing device receives an input in a graphical user interface (GUI) from a human interface device (HID), and a screen location of the HID input, and a user identifier; program instructions to identify, by the computer, an active window in the received screenshot, based on the screen location of the HID input; program instructions to extract, by the computer, from the screenshot, the identified active window; program instructions to determine, by the computer, a textual title of the active window; and program instructions to store one or more electronic records, by the computer, comprising the extracted active window, the textual title of the active window, and the user identifier to a data store.

14 . A computer system according to claim 13 , wherein program instructions to identify the active window comprises:
program instructions to locate edges of one or more rectangles present in the received screenshot; program instructions to identify one or more rectangles corresponding to the located edges; program instructions to detect one or more windows corresponding to the identified rectangles; and program instructions to identify the active window based on the screen location of the HID input within the detected one or more windows.

15 . A computer system according to claim 13 , wherein program instructions to determine the textual title of the active window comprises:
program instructions to recognize characters in a general region of a title bar of the active window using an object character recognition (OCR) technique; and program instructions to extract the characters to develop the textual title of the active window.

16 . A computer system according to claim 13 ,
wherein the program instructions to receive a screenshot of a display device further comprises program instructions to receive one or more of: a device identifier, a network identifier, an active user program identifier, raw HID inputs, and a date/time stamp; and wherein the program instructions to store one or more electronic records further comprises program instructions to store one or more of: the device identifier, the network identifier, the active user program identifier, the raw H

Assignees

• IBM

Inventors

• Li Linlin
• Soenaryo Edwin B

Classifications

• G06V30/10

  Character recognition · CPC title

• G06T7/13

  Edge detection · CPC title

• G06F11/3476

  Data logging (G06F11/14, G06F11/2205 take precedence) · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06F16/24578

  using ranking · CPC title