Utilizing reinforcement learning for goal oriented website navigation

Patent metadata

Field	Value
Publication number	US-11983227-B2
Application number	US-202217891894-A
Country	US
Kind code	B2
Filing date	Aug 19, 2022
Priority date	Aug 6, 2019
Publication date	May 14, 2024
Grant date	May 14, 2024

Abstract

A computer system receives a goal for an environment, wherein the environment corresponds to at least one webpage. The computer system receives one or more classifiers corresponding to the environment, wherein the one or more classifiers provide information corresponding to a current webpage and information corresponding to one or more previous actions taken by a web crawler. The computer system identifies a recommended next action based on the one or more classifiers. The computer system transmits the recommended next action to the web crawler to cause the web crawler to perform the recommended next action.

First claim

What is claimed is:

1. A method, comprising:
accessing webpage navigation data pertaining to one or more webpages, wherein the webpage navigation data has been extracted by an automated software application from the one or more webpages;
identifying, based on the webpage navigation data, one or more classifiers corresponding to a state of an environment that is usable by the automated software application to interpret a current environment of to the one or more webpages such that information corresponding to a type of the one or more webpages and a navigation history of the one or more webpages are extractable, and wherein the one or more classifiers further indicate a number of clicks or an action taken by visitors of the one or more webpages;
determining, based on the one or more classifiers, a recommended next action for the automated software application; and
causing the automated software application to perform the recommended next action;
wherein one or more of the accessing, the identifying, the determining, or the causing are performed by a server that includes one or more hardware processors.

2. The method of claim 1 , wherein the automated software application comprises a web crawler that is configured to extract the webpage navigation data from the one or more webpages at least in part by crawling the one or more webpages without direct human input, and wherein the one or more classifiers further indicate an action taken by the web crawler.

3. The method of claim 2 , wherein the web crawler is further configured to extract the webpage navigation data at least in part by accessing a source code associated with the one or more webpages.

4. The method of claim 2 , wherein the web crawler is further configured to extract the webpage navigation data at least in part by accessing a uniform resource locator (URL) associated with the one or more webpages, hypertext markup language (HTML) elements associated with the one or more webpages, or metadata associated with the one or more webpages.

5. The method of claim 1 , wherein the one or more classifiers are identified at least in part via a machine learning process.

6. The method of claim 5 , wherein the machine learning process comprises using a machine learning model trained at least in part by instructing the automated software application to traverse the one or more webpages using different navigation paths.

7. The method of claim 1 , wherein the one or more classifiers each indicates whether the one or more webpages is a product listing page or a checkout page of an online merchant, or whether a digital shopping cart contains any items.

8. The method of claim 1 , wherein the navigation history comprises one or more previous actions taken by the automated software application.

9. The method of claim 1 , further comprising inputting the one or more classifiers into a reinforcement learning model, wherein the recommended next action is determined further at least in part based on the reinforcement learning model.

10. The method of claim 9 , further comprising:
determining whether an actual next action taken by the automated software application corresponds to a predefined reward; and
updating the reinforcement learning model based on the determining of whether the actual next action taken by the automated software application corresponds to the predefined reward.

11. A server system, comprising:
a processor; and
a non-transitory computer-readable medium having stored thereon instructions that are executable by the processor to cause the server system to perform operations comprising:
causing a web crawler software to electronically crawl one or more webpages;
extracting, at least in part via a crawling of the one or more webpages by the web crawler software, webpage navigation data associated with the one or more webpages;
determining, at least in part via the webpage navigation data, classifier information corresponding to a type of the one or more webpages and a navigation history of the one or more webpages, the classifier information further detailing a state of an environment that is usable by the web crawler software to interpret a current environment of the one or more webpages;
determining a number of clicks or one or more actions taken by one or more visitors of the one or more webpages; and
providing, based on the classifier information and the determined number of clicks or the one or more actions taken, a recommendation to the web crawler software to perform an action.

12. The server system of claim 11 , wherein the causing the web crawler software to crawl the one or more webpages comprises causing the web crawler software to crawl a source code of the one or more webpages.

13. The server system of claim 11 , wherein the causing the web crawler software to crawl the one or more webpages comprises causing the web crawler software to access a uniform resource locator (URL) associated with the one or more webpages, hypertext markup language (HTML) elements associated with the one or more webpages, or metadata associated with the one or more webpages.

14. The server system of claim 11 , wherein the determining the classifier information comprises performing a machine learning process.

15. The server system of claim 14 , wherein the machine learning process is performed at least in part by using a machine learning model trained by instructing the web crawler software to traverse the one or more webpages using different navigation paths.

16. The server system of claim 11 , wherein:
the classifier information corresponding to the type of the one or more webpages comprises whether the one or more webpages include a product listing page or a checkout page of an online merchant; or
the classifier information corresponding to the navigation history comprises one or more previous actions taken by the web crawler software.

17. The server system of claim 11 , wherein the operations further comprise:
determining whether a previous action taken by the web crawler software corresponded to a predefined reward; and
updating a reinforcement learning model based on whether the previous action taken by the web crawler software corresponded to the predefined reward, wherein the providing the recommendation is based on the reinforcement learning model.

18. A method, comprising:
causing a web crawler software application to crawl one or more webpages;
accessing, based on results obtained by the web crawler software application, webpage navigation data associated with the one or more webpages;
accessing, based on the webpage navigation data, classifier information detailing a state of an environment that is usable by the web crawler software application to interpret a current environment of the one or more webpages;
identifying, based on the classifier information, one or more previous actions taken by the web crawler software application during a previous traversal through the one or more webpages, or a number of clicks or one or more actions taken by one or more visitors of the one or more webpages;
determining, based on at least a portion of the classifier information that identifies the number of clicks or the one or more actions taken by the one or more visitors, one or more next recommended action to be performed by the web crawler software application; and
providing the one or more next recommended actions to the web crawler software application.

19. The method of claim 18 , wherein the identifying is performed at least in part by using a trained machine learning

Assignees

• Paypal Inc

Inventors

• Cohen Yaeli
• Cohen Adam
• Raisken Yarden

Classifications

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title