Identifying web elements based on user browsing activity and machine learning

Patent metadata

Field	Value
Publication number	US-12455929-B2
Application number	US-202318535066-A
Country	US
Kind code	B2
Filing date	Dec 11, 2023
Priority date	Apr 18, 2019
Publication date	Oct 28, 2025
Grant date	Oct 28, 2025

Abstract

A computer-implemented method includes tracking, by a computing device, user browsing activity of a first page having known elements; mapping, by the computing device, the user browsing activity to the known elements; storing, by the computing device, mapping information that maps the user browsing activity to the known elements; tracking, by the computing device, user browsing activity of a second page having unknown elements; identifying, by the computing device, the unknown elements based on the mapping information and the user browsing activity of the second page; and executing, by the computing device, one or more computer-based instructions based on the determining the unknown elements that were identified.

First claim

What is claimed is:

1. A computer-implemented method comprising:
obtaining, by a at least one processor, present user activity with a present user graphical interface of an application, the present graphical interface comprising at least one user interface (UI) element, the at least one UI element being a digital object in the graphical interface;
determining, by the at least one processor, based on the present user activity, a presence of at least one untested user interface (UI) element in the present user graphical interface of the application;
applying, by the at least one processor, a mapping data model to the present user activity to identify at least one UI element type associated with the at least one untested UI element so as to output at least one identified UI element of the present user graphical interface of the application;
wherein the mapping data model is trained to correlate between historical user activity and at least one pre-tested element of at least one previously-visited user graphical interface page; and

triggering, by the at least one processor, at least one custom test to be performed with the present user graphical interface of the application, the at least one custom test comprising one or more computer-based instructions configured to perform at least one operation to test the at least one identified UI element based at least in part on the at least one UI element type.

2. The method of claim 1 , further comprising:
utilizing, by the at least one processor, the mapping data model to identify at least one web element attribute of the at least one pre-tested UI element; and
wherein the at least one pre-tested element comprises at least one pre-tested web element attribute.

3. The method of claim 2 , further comprising:
determining, by the at least one processor, at least one testing parameter based at least in part on the at least one web element attribute; and
generating, by the at least one processor, the at least one custom test based at least in part on the at least one testing parameter.

4. The method of claim 2 , wherein the at least one web element attribute includes at least one of:
a type;
a size;
a shape; or
a page placement location.

5. The method of claim 1 , further comprising:
utilizing, by the at least one processor, the mapping data model to identify at least one type associated with the at least one pre-tested UI element; and
generating, by the at least one processor, the at least one custom test based at least in part on at least one type.

6. The method of claim 1 , wherein the custom test is based on information stored by a test suite server.

7. The method of claim 1 , wherein the user activity includes at least one from the group consisting of:
a mouse movement;
a cursor movement;
a cursor hover;
a mouse click;
a keystroke;
a page selection;
a page scrolling;
a page zooming;
a page viewing duration;
a page text;
a browser type; and
a geographic location.

8. The method of claim 1 , wherein the mapping data model includes at least one selected from the group consisting of:
training a classifier; and
inputting the user activity into a neural network.

9. The method of claim 1 , wherein the user activity comprises at least one browsing activity input.

10. The method of claim 9 , further comprising:
tracking, by the at least one processor, the user activity on the present user graphical interface having the at least one pre-tested UI element;
mapping, by the at least one processor, the at least one browsing activity input to the at least one pre-tested UI elements based at least in part on the user activity at the user at least one processor to the at least one pre-tested UI element; and
retraining, by the at least one processor, the mapping data model to recognize the UI elements from the user activity based at least in part on the mapping of the at least one browsing activity input to the at least one pre-tested UI element.

11. A system comprising:
at least one processor configured to execute software instructions, wherein the software instructions, upon execution, cause the at least one processor to:
obtain present user activity with a present user graphical interface of an application, the present graphical interface comprising at least one user interface (UI) element, the at least one UI element being a digital object in the graphical interface;
determine based on the present user activity, a presence of at least one untested user-interface (UI) element in the present user graphical interface of the application;
apply a mapping data model to the present user activity to identify the at least one untested UI element as at least one pre-tested UI element to output at least one identified UI element of the present user graphical interface of the application;
wherein the mapping data model is trained to correlate between historical user activity and the at least one pre-tested UI element of at least one previously-visited user graphical interface page; and

trigger at least one custom test to be performed with the present user graphical interface of the application, the at least one custom test comprising one or more computer-based instructions configured to perform at least one operation to test the at least one identified UI element.

12. The system of claim 11 , wherein the at least one processor is further configured to execute the software instructions that, upon execution, further cause the at least one processor to:
utilize the mapping data model to identify at least one web element attribute of the at least one pre-tested UI element; and
wherein the at least one pre-tested UI element comprises at least one pre-tested web element attribute.

13. The system of claim 12 , wherein the at least one processor is further configured to execute the software instructions that, upon execution, further cause the at least one processor to:
determine at least one testing parameter based at least in part on the at least one web element attribute; and
generate the at least one custom test based at least in part on the at least one testing parameter.

14. The system of claim 12 , wherein the at least one web element attribute includes at least one of:
a type;
a size;
a shape; or
a page placement location.

15. The system of claim 11 , wherein the at least one processor is further configured to execute the software instructions that, upon execution, further cause the at least one processor to:
utilize the mapping data model to identify at least one type associated with the at least one pre-tested UI element; and
generate the at least one custom test based at least in part on at least one type.

16. The system of claim 11 , wherein the custom test is based on information stored by a test suite server.

17. The system of claim 11 , wherein the user activity includes at least one from the group consisting of:
a mouse movement;
a cursor movement;
a cursor hover;
a mouse click;
a keystroke;
a page selection;
a page scrolling;
a page zooming;
a page viewing duration;
a page text;
a browser type; and
a geographic location.

18. The system of claim 11 , wherein the mapping data model includes at least one selected from the group consisting of:
training a classifier; and
inputting the user activity into a neural network.

19. The system of claim 11 , wherein the user activity comprises at least one browsing activity input.

20. The system of claim 19 , wherein the at least one processor is

Assignees

• Capital One Services Llc

Inventors

• Walters Austin
• Goodsitt Jeremy
• Pham Vincent
• Watson Mark
• Taghi Abad Fardin Abdi
• Farivar Reza
• Truong Anh
• Taylor Kenneth

Classifications

• G06N20/00

  Machine learning · CPC title

• G06F11/3438 Primary

  monitoring of user actions (tracking the activity of the user H04L67/535) · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/0499

  Feedforward networks · CPC title

• G06N3/09

  Supervised learning · CPC title