Configuring web crawler to extract web page information

Patent metadata

Field	Value
Publication number	US-9015144-B2
Application number	US-201314081105-A
Country	US
Kind code	B2
Filing date	Nov 15, 2013
Priority date	Jul 22, 2011
Publication date	Apr 21, 2015
Grant date	Apr 21, 2015

Abstract

Web crawling configuration includes: obtaining a webpage comprising a plurality of receiving a user selection of a node in the webpage; presenting a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node, the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising: a first option to perform a first web crawling action in the event that the node include a first type of the element; and a second option to perform a second web crawling action in the event that the node includes a second type of the element; receiving a user input specifying the web crawling configuration option; and storing user specified web crawling configuration option, performing the web crawling action on the node according to the user input, or both.

First claim

What is claimed is:

1. A system for configuring web crawling, comprising:
one or more processors to:
obtain a webpage comprising a plurality of nodes;
present the webpage to a user;
receive a user selection of a node in the webpage, the node comprising at least one element;
in response to the user selection of the node, present a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node, the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising:
a first option to perform a first web crawling action in the event that the node include a first type of the element; and
a second option to perform a second web crawling action in the event that the node includes a second type of the element, the first web crawling action being different from the second web crawling action, and the first type of the element being different from the second type of the element;

receive a user input specifying a selected one of the set of web crawling configuration option pertaining to the web crawling action to be performed with respect to the node; and
store the user specified web crawling configuration option, perform the corresponding web crawling action on the node according to the user input, or both; and

one or more memories coupled to the one or more processors, to provide the one or more processors with instructions.

2. The system of claim 1 , wherein the one or more processors are further to receive another user selection of a second node in the webpage, and to receive a user input specifying a web crawling configuration option pertaining to a web crawling action to be performed with respect to the second node.

3. The system of claim 1 , wherein the one or more processors are further to process the obtained webpage such that each node is configurable for web crawling.

4. The system of claim 1 , wherein the one or more processors are further to present a preview of the performed web crawling action according to the user input.

5. The system of claim 1 , wherein the webpage is presented to the user via a graphical user interface (GUI).

6. The system of claim 1 , wherein the first type of the element corresponds to text and the first web crawling action comprises text extraction.

7. The system of claim 6 , wherein the web crawling configuration options for text to extraction comprise configurations for one or more of: a filter, address extraction, date extraction, extracted text name, extracted text description, and format.

8. The system of claim 1 , the one or more processors are further to receive a user indication that the option pertaining to the web crawling action is to be applied to multiple nodes.

9. The system of claim 1 , the one or more processors are further to receive a user indication is that the user specified web crawling configuration option is to be applied to multiple nodes.

10. The system of claim 1 , wherein the first type of the element corresponds to a link and the first web crawling action comprises downloading the link.

11. The system of claim 10 , wherein downloading the link comprises obtaining a subpage; and the one or more processors are further to present an additional web crawling configuration option pertaining to a web crawling action to be performed with respect to one or more nodes on the subpage.

12. The system of claim 11 , the one or more processors are further to present a navigation tree of webpages, and associated web crawling configuration options with the webpage in the navigation tree of webpages.

13. The system of claim 1 , wherein the first type of the element corresponds to a link and the first web crawling action comprises one or more of the following link processing settings: extracting link text, extracting alt text, and extracting link target URL.

14. The system of claim 1 , wherein the first type of the element corresponds to an image and the first web crawling action comprises a setting for downloading an image, capturing the image, or both.

15. The system of claim 1 , wherein the web crawling configuration option comprises a setting for an event associated with the node, wherein the event includes: click, double click, mouse hover, rolling wheel operation, or mouse exit.

16. The system of claim 1 , wherein the web crawling configuration option comprises a setting for a form element comprising a text box, wherein the setting comprises input text.

17. The system of claim 1 , wherein the web crawling configuration option comprises a setting for a form element comprising: a text box, a button, a radio button, or a check box.

18. The system of claim 1 , the one or more processors are further to store a locator of the is node.

19. The system of claim 18 , wherein the locator of the node selected by the user is based on XPath.

20. The system of claim 1 , wherein performing the web crawling action on the node comprises downloading the webpage to be crawled.

21. The system of claim 1 , wherein performing the web crawling action on the node comprises rendering Javascript code of at least a portion of the webpage.

22. The system of claim 1 , further comprising storing results of the performed web crawling action.

23. A method for configuring web crawling, comprising:
obtaining, using one or more computer processors, a webpage comprising a plurality of nodes;
presenting the webpage to a user;
receiving a user selection of a node in the webpage, the node comprising at least one element;
in response to the user selection of the node, presenting a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node, the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising:
a first option to perform a first web crawling action in the event that the node include a first type of the element; and
a second option to perform a second web crawling action in the event that the node includes a second type of the element, the first web crawling action being different from the second web crawling action, and the first type of the element being different from the second type of the element;

receiving a user input specifying the web crawling configuration option pertaining to the web crawling action to be performed with respect to the node; and
storing user specified web crawling configuration option, performing the web crawling action on the node according to the user input, or both.

24. A tangible non-transitory computer readable storage medium, having a computer program product for configuring web crawling embodied therein comprising computer instructions for:
obtaining a webpage comprising a plurality of nodes;
presenting the webpage to a user;
receiving a user selection of a node in the webpage, the node comprising at least one element;
in response to the user selection of the node,
presenting a set of web crawling configuration options pertaining to a web crawling action to be performed with respect to the node,
the set of web crawling configuration options depending at least in part on a type of an element included in the node and comprising:

a first option to perform a first web crawling action in the event that the node include a first type of the element; and
a second option to perform a second web crawling action in the event that the

Assignees

• Alibaba Group Holding Ltd

Inventors

• Sun Yiming
• Qiang Qi
• Cai Boyang
• Jin Xiaojun
• Wu Zongyuan

Classifications

• G06Q10/10

  Office automation; Time management · CPC title

• G06F17/30864 Primary

  Physics · mapped topic

• G06Q30/02

  Marketing; Price estimation or determination; Fundraising · CPC title

• G06F17/30873

  Physics · mapped topic

• G06F17/30707

  Physics · mapped topic