Automating interactions with software user interfaces

Patent metadata

Field	Value
Publication number	US-9740506-B2
Application number	US-77201310-A
Country	US
Kind code	B2
Filing date	Apr 30, 2010
Priority date	Nov 21, 2002
Publication date	Aug 22, 2017
Grant date	Aug 22, 2017

Abstract

A method and apparatus applies an action to a software application by determining a target object for the input action. The determination of the target object is performed by identifying the target object through socially identifying object information relative to a reference object. Then, the input action is applied to the target object.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining a target object in a graphical user interface (GUI) for a software application input action based upon socially identifying information of the target object, the socially identifying information describing the target objects relationship to one or more peer objects of a same type as the target object and/or descendent objects in the GUI and comprising:
a type of the target object's parent,
a rank of the target object relative to the one or more peer objects of the same type as the target object, wherein the rank indicates a location of the target object in the GUI relative to the one or more peer objects of the same type as the target object, and
a location of a label of the target object, wherein the label is separate from the target object and the location of the label describes a position of the label in the GUI relative to the target object; and

applying the input action to the target object.

2. The method of claim 1 , wherein the target object is further determined using name information.

3. The method of claim 2 , wherein the name information comprises an object name.

4. The method of claim 1 , wherein the socially identifying information further comprises information about a parent object to the target object.

5. The method of claim 1 , wherein the socially identifying information includes an identifier of which of a number of unnamed objects the target object is.

6. The method of claim 1 , wherein the input action is provided by a script file, the method further comprising generating the script file by recording one or more user actions with respect to the GUI.

7. The method of claim 1 , wherein the socially identifying information does not identify a name of the target object.

8. The method of claim 1 , wherein the socially identifying information further comprises a spatial location, the spatial location comprising a numerical value indicating that the one or more objects are spatially located to the left or right of the target object.

9. The method of claim 1 , wherein the socially identifying information further comprises a spatial location, the spatial location comprising a numerical value indicating that the one or more objects are spatially located above or below the target object.

10. One or more tangible computer-readable storage media storing computer-readable instructions that when executed by a computer, cause the computer to perform a method, the method comprising:
determining a target object in a graphical user interface (GUI) for an action, the determining being based upon data defined in a script file and describing the target object's relationship to one or more peer objects in the GUI and comprising:
a type of the target object's parent,
a rank of the target object relative to the one or more peer objects of a same type as the target object, wherein the rank is determined by signature information defined in the script file and is based on a relative location of the one or more peer objects with respect to the target object in the GUI, and
a location of a label of the target object, wherein the label is separate from the target object and the location of the label describes a position of the label in the GUI relative to the target object; and

applying the action to the target object, the action being defined in the script file.

11. The one or more tangible computer-readable storage media of claim 10 , wherein the target object is further determined using name information.

12. The one or more tangible computer-readable storage media of claim 11 , wherein the name information comprises an object name.

13. The one or more tangible computer-readable storage media of claim 11 , wherein the name information comprises an identifier of a parent object to the target object.

14. The one or more tangible computer-readable storage media of claim 10 , wherein the data further comprises information about a parent object to the target object.

15. The one or more tangible computer-readable storage media of claim 10 , wherein the data further comprises an identifier of which of a number of unnamed objects the target object is.

16. The one or more tangible computer-readable storage media of claim 10 , wherein determining a target object is performed by interacting with a java virtual machine.

17. The one or more tangible computer-readable storage media of claim 10 wherein the target object is a java component.

18. The one or more tangible computer-readable storage media of claim 17 , wherein the java component comprises a top level container name and a component name.

19. An apparatus comprising:
a processor; and
a computer-readable medium having stored thereon a plurality of instructions which, when executed, cause the processor to:
determine a target object in a graphical user interface (GUI) for the input action based upon socially identifying information of the target object, the socially identifying information describing hierarchy information about the target object's relationship to one or more peer objects in the GUI, wherein the one or more peer objects have a same type as the target object, and a rank of the target object relative to the one or more peer objects having the same type as the target object, wherein the rank is based on a relative location of the one or more peer objects with respect to the target object in the GUI, wherein the socially identifying information further comprises label signature information, the label signature information describing a location of a label of the target object, wherein the label is separate from the target object and the location of the label describes a position of the label in the GUI relative to the target object; and

apply the input action to the target object, the input action being defined in a script file including the hierarchy information.

20. The apparatus of claim 19 , wherein the target object is further determined using name information.

21. The apparatus of claim 20 , wherein the name information comprises one or more of an object name or an identifier of a parent object to the target object.

22. The apparatus of claim 19 , wherein the socially identifying information comprises information about a parent object to the target object.

23. The apparatus of claim 19 , wherein the socially identifying information further comprises an identifier of which of a number of unnamed objects the target object is.

24. One or more tangible computer-readable storage media storing computer-readable instructions that when executed by a computer, cause the computer to perform a method, the method comprising:
determining a target object in a graphical user interface (GUI) for an action, the determining being based upon socially identifying information of the target object, the socially identifying information being defined in a script file and describing hierarchy information about the target object's relationship to one or more peer and/or descendent objects in the GUI and comprising a name of the target objects parent and rank of the target object relative to the one or more peer objects and/or descendent objects in the GUI, wherein the rank is based on a relative location of a label component to the target object, wherein the label component is one of the peer objects and/or descendent objects, wherein the relative location is determined using signature information defined in the script file, whe

Assignees

• Ren Bing
• Mentor Graphics Corp

Inventors

• Ren Bing

Classifications

• G06F9/4443 Primary

  Physics · mapped topic

• G06F9/451 Primary

  Execution arrangements for user interfaces · CPC title