Automatic crawling of encoded dynamic URLs

Patent metadata

Field	Value
Publication number	US-9754028-B2
Application number	US-201213459536-A
Country	US
Kind code	B2
Filing date	Apr 30, 2012
Priority date	Oct 11, 2011
Publication date	Sep 5, 2017
Grant date	Sep 5, 2017

Abstract

A method includes retrieving navigational state information corresponding to a URL; analyzing which navigational state to consider for page identification; comparing the navigational state information to previously stored navigational state information corresponding to one or more previously visited URLs; determining if the URL has been previously visited; and retrieving content associated with the URL if the URL has not been previously visited.

First claim

What is claimed is:

1. A method, comprising:
Receiving, using a computing device, an encoded URL corresponding to a web-portal page hosted by a web-portal server, wherein a text of the encoded URL does not provide information about content of the web-portal page, the content comprising a plurality of re-arrangeable portlets, and wherein the web-portal server provides, via a representational state transfer service, to a web-crawler, a navigational state information of the web-portal page using a predetermined format;
initializing a list of parameters, wherein the list identifies that the web-portal page corresponding to the encoded URL has been visited, and determining the parameters to add to the list by:
sending the encoded URL to the representational state transfer service from the web-portal server hosting the web-portal page for decoding the encoded URL;
receiving the navigational state information of the web-portal page from the representational state transfer service, wherein the navigational state information is decoded from the encoded URL, and comprises a selection-node-ID and a resource-ID, the selection-node-ID being an identifier assigned to the web-portal page, and the resource-ID being an identifier associated with a resource used by the web-portal page, wherein,
determining a type of the encoded URL based on the received representational state information, and

when the type is determined to be a resource URL, storing the resource-ID in the list of parameters; and
when the type is determined to be an engine URL, storing the selection-node-ID in the list of parameters, and
when the type is determined to be a portlet, storing information of the portlet in the list of parameters;

determining if the encoded URL has been previously visited based on the list of parameters; and
retrieving the content associated with the encoded URL if the encoded URL has not been previously visited.

2. The method of claim 1 , further comprising: analyzing content associated with the encoded URL; and exploring an additional URL found in the content associated with the encoded URL.

3. The method of claim 2 , further comprising detecting if the additional
URL is a logout link prior to exploring the additional URL.

4. The method of claim 1 , wherein the navigational state information from the representational state transfer service is received in XML format.

5. The method of claim 1 , wherein the navigational state information includes information indicative of the type of the encoded URL.

6. The method of claim 1 , wherein the web-portal server uses websphere portal framework.

Assignees

• Brake Nevon C
• Islam Obidul
• Sharabani Adi
• IBM

Inventors

• Brake Nevon C
• Islam Obidul
• Sharabani Adi

Classifications

• G06F17/30864 Primary

  Physics · mapped topic

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title