Method and apparatus to throttle media access by web crawlers

Patent metadata

Field	Value
Publication number	US-2016127262-A1
Application number	US-201414530659-A
Country	US
Kind code	A1
Filing date	Oct 31, 2014
Priority date	Oct 31, 2014
Publication date	May 5, 2016
Grant date	—

Abstract

Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example method disclosed herein includes obtaining, at a server, a media request message for media hosted by the server, the media request message requesting access to the media, characterizing a media-requesting source associated with the media request message, and inserting a time delay in a media response message to the media-requesting source based on the characterization.

First claim

What is claimed is:

1 . A method to throttle media access at a server, the method comprising:
obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media; characterizing a media-requesting source associated with the media request message; and inserting a time delay in a media response message to the media-requesting source based on the characterization.

2 . A method as defined in claim 1 , wherein characterizing the media-requesting source comprises:
extracting source-identifying information from the media request message; and matching the source-identifying information with a source category identifier.

3 . A method as defined in claim 2 further comprising obtaining the source category identifier from a data collection entity.

4 . A method as defined in claim 2 further comprising identifying the source category identifier in a lookup table with the source-identifying information

5 . A method as defined in claim 1 , wherein the media-requesting source is a web crawler.

6 . A method as defined in claim 1 , wherein inserting the time delay in the media response message comprises including a delay tag in the media response message.

7 . A method as defined in claim 6 , wherein the delay tag includes executable instructions.

8 . A method as defined in claim 7 , wherein the executable instructions include a native delay function.

9 . A method as defined in claim 6 , wherein executing the delay tag generates the time delay.

10 . A method as defined in claim 6 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.

11 . A method as defined in claim 1 , wherein inserting the time delay in the media response message comprises selecting a pre-existing copy of the media with a delay tag associated with the time delay.

12 . A method as defined in claim 1 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

13 . A system comprising:
a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media; a category handler to characterize a media-requesting source associated with the media request message; and a penalty manager to insert a time delay in a media response message to the media-requesting source based on the characterization.

14 . A system as defined in claim 13 , wherein the category handler is to:
extract source-identifying information from the media request message; and match the source-identifying information with a source category identifier.

15 . A system as defined in claim 14 , wherein the category handler is to obtain the source category identifier from a data collection entity.

16 . A system as defined in claim 14 , wherein the category handler is to identify the source category identifier in a lookup table with the source-identifying information.

17 . A system as defined in claim 13 , wherein the media-requesting source is a web crawler.

18 . A system as defined in claim 13 , wherein the penalty manager is to include a delay tag in the media response message.

19 . A system as defined in claim 18 , wherein the delay tag includes executable instructions.

20 . A system as defined in claim 19 , wherein the executable instructions include a native delay function.

21 . A system as defined in claim 18 , wherein execution of the delay tag generates the time delay.

22 . A system as defined in claim 18 , wherein execution of the delay tag requests data from a web site associated with generating the time delay.

23 . A system as defined in claim 13 , wherein the penalty manager is to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay

24 . A system as defined in claim 13 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

25 . A tangible computer readable storage medium comprising instructions that, when executed, cause a server hosting media to at least:
obtain a media request message for media hosted by the server, the media request message requesting access to the media; characterize a media-requesting source associated with the media request message; and insert a time delay in a media response message to the media-requesting source based on the characterization.

26 . A tangible computer readable storage medium as defined in claim 25 , wherein the instructions further cause the server to characterize the media-requesting source by:
extracting source-identifying information from the media request message; and matching the source-identifying information with a source category identifier.

27 . A tangible computer readable storage medium as defined in claim 26 , wherein the instructions further cause the server to identify the source category identifier in a lookup table with the source-identifying information

28 . A tangible computer readable storage medium as defined in claim 25 , wherein the instructions further cause the server to insert the time delay in the media response message by including a delay tag in the media response message.

29 . A tangible computer readable storage medium as defined in claim 25 , wherein the instructions further cause the server to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.

Assignees

• Nielsen Co Us Llc

Inventors

• Lawrence Iii Bradley R

Classifications

• H04L47/24

  Traffic characterised by specific attributes, e.g. priority or QoS · CPC title

• H04L47/80 Primary

  Actions related to the user profile or the type of traffic · CPC title

• H04L47/32

  by discarding or delaying data units, e.g. packets or frames · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title