Configuring web crawler to extract web page information

Patent metadata

Field	Value
Publication number	US-9330179-B2
Application number	US-201414523731-A
Country	US
Kind code	B2
Filing date	Oct 24, 2014
Priority date	Jul 22, 2011
Publication date	May 3, 2016
Grant date	May 3, 2016

Abstract

Web crawling configuration includes: obtaining a webpage comprising a plurality of receiving a user selection of a node in the webpage; presenting a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node, the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising: a first option to perform a first web crawling action in the event that the node include a first type of the element; and a second option to perform a second web crawling action in the event that the node includes a second type of the element; receiving a user input specifying the web crawling configuration option; and storing user specified web crawling configuration option, performing the web crawling action on the node according to the user input, or both.

First claim

What is claimed is:

1. A system for configuring web crawling, comprising:
one or more processors to:
obtain a webpage comprising a plurality of nodes;

present the webpage to a user;
receive a user selection of a node in the webpage, the node comprising at least one element;
in response to the user selection of the node, present a web crawling configuration option pertaining to a web crawling action to be performed with respect to the node,
the web crawling configuration option depending at least in part on a type of an element included in the node;
receive a user input specifying the web crawling configuration options pertaining to the web crawling action to be performed with respect to the node and a user indication to specify repeatedly the web crawling action to be performed on the plurality of nodes;
specify repeatedly the web crawling action to be performed for the plurality of nodes;
store user specified web crawling configuration options;
perform the web crawling action on the plurality of nodes according to the user input;
receive another user selection of a second node in the webpage; and
receive a user input specifying an option pertaining to a web crawling action to be performed with respect to the second node; and
one or more memories coupled to the one or more processors, to provide the one or more processors with instructions.

2. The system of claim 1 , wherein the one or more processors are further to process the obtained webpage such that each node is configurable for web crawling.

3. The system of claim 1 , wherein the one or more processors are further to present a preview of the performed web crawling action according to the user input.

4. The system of claim 1 , wherein the webpage is presented to the user via a graphical user interface (GUI).

5. The system of claim 1 , wherein the web crawling action comprises text extraction.

6. The system of claim 5 , wherein the web crawling configuration options for text extraction comprise configurations for one or more of: a filter, address extraction, date extraction, extracted text name, extracted text description, and format.

7. The system of claim 1 , wherein the web crawling configuration option comprises settings associated with a node that is a link and the web crawling action comprises downloading the link.

8. The system of claim 7 , wherein downloading the link comprises obtaining a subpage; and the one or more processors are further to present a web crawling configuration option pertaining to a web crawling action to be performed with respect to one or more nodes on the subpage.

9. The system of claim 8 , the one or more processors are further to present a navigation tree of webpages, and associated web crawling configuration options with the webpage in the navigation tree of webpages.

10. The system of claim 1 , wherein the web crawling configuration option pertaining to the web crawling action comprises one or more of the following link processing settings: extracting link text, extracting alt text, and extracting link target URL.

11. The system of claim 1 , wherein the web crawling configuration option comprises a setting for downloading an image, capturing the image, or both.

12. The system of claim 1 , wherein the web crawling configuration option comprises a setting for an event associated with the node, wherein the event includes: click, double click, mouse hover, rolling wheel operation, or mouse exit.

13. The system of claim 1 , wherein the web crawling configuration option comprises a setting for a form element comprising a text box, wherein the setting comprises input text.

14. The system of claim 1 , wherein the web crawling configuration option comprises a setting for a form element comprising: a text box, a button, a radio button, or a check box.

15. The system of claim 1 , the one or more processors are further to store a locator of the node.

16. The system of claim 15 , wherein the locator of the node selected by the user is based on XPath.

17. The system of claim 1 , wherein performing the web crawling action on the node comprises downloading the webpage to be crawled.

18. The system of claim 1 , wherein performing the web crawling action on the node comprises rendering Javascript code of at least a portion of the webpage.

19. The system of claim 1 , further comprising storing results of the performed web crawling action.

20. A method for configuring web crawling, comprising:
obtaining, using one or more computer processors, a webpage comprising a plurality of nodes;
presenting the webpage to a user;
receiving a user selection of a node in the webpage, the node comprising at least one element;
in response to the user selection of the node,
presenting a web crawling configuration option pertaining to a web crawling action to be performed with respect to the node, the web crawling configuration option depending at least in part on a type of an element included in the node;
receiving a user input specifying the web crawling configuration options pertaining to the web crawling action to be performed with respect to the node and a user indication to specify repeatedly the web crawling action to be performed on the plurality of nodes;
specifying repeatedly the web crawling action to be performed for the plurality of nodes:
storing user specified web crawling configuration options;
performing the web crawling action on the plurality of nodes according to the user input;
receiving another user selection of a second node in the webpage; and
receiving a user input specifying an option pertaining to a web crawling action to be performed with respect to the second node.

21. A computer program product for configuring web crawling, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for:
obtaining, using one or more computer processors, a webpage comprising a plurality of nodes;
presenting the webpage to a user;
receiving a user selection of a node in the webpage, the node comprising at least one element;
in response to the user selection of the node, presenting a web crawling configuration option pertaining to a web crawling action to be performed with respect to the node, the web crawling configuration option depending at least in part on a type of an element included in the node;
receiving a user input specifying the web crawling configuration options pertaining to the web crawling action to be performed with respect to the node and a user indication to specify repeatedly the web crawling action to be performed on the plurality of nodes;
specifying repeatedly the web crawling action to be performed for the plurality of nodes.
storing user specified web crawling configuration options;
performing the web crawling action on the plurality of nodes according to the user input;
receiving another user selection of a second node in the webpage; and
receiving a user input specifying an option pertaining to a web crawling action to be performed with respect to the second node.

Assignees

• Alibaba Group Holding Ltd

Inventors

• Sun Yiming
• Qiang Qi
• Cai Boyang
• Jin Xiaojun
• Wu Zongyuan

Classifications

• G06F17/30864 Primary

  Physics · mapped topic

• G06F17/30905

  Physics · mapped topic

• G06F17/30887

  Physics · mapped topic

• G06F17/30707

  Physics · mapped topic

• G06Q10/10

  Office automation; Time management · CPC title