Method and apparatus to throttle media access by web crawlers

Patent metadata

Field	Value
Publication number	US-10257113-B2
Application number	US-201815889173-A
Country	US
Kind code	B2
Filing date	Feb 5, 2018
Priority date	Oct 31, 2014
Publication date	Apr 9, 2019
Grant date	Apr 9, 2019

Abstract

Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example apparatus disclosed herein includes a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media. The example apparatus also includes a category handler to determine a source indicator of a media-requesting source associated with the media request message, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider. The example apparatus also includes a penalty manager to insert a time delay including a delay tag in the media response message to the media-requesting source based on the source indicator, the delay tag to cause a delay time period based on the source indicator, at least one of the data interface, category handler, and penalty manager is implemented on a logic circuit.

First claim

What is claimed is:

1. An apparatus comprising:
a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media;
a category handler to determine a source indicator of a media-requesting source associated with the media request message, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider; and
a penalty manager to insert a time delay including a delay tag in the media response message to the media-requesting source based on the source indicator, the delay tag to cause a delay time period based on the source indicator, at least one of the data interface, category handler, and penalty manager is implemented on a logic circuit.

2. The apparatus as defined in claim 1 , wherein the media-requesting source is a web crawler.

3. The apparatus as defined in claim 1 , wherein the delay tag includes executable instructions.

4. The apparatus as defined in claim 3 , wherein the executable instructions include a native delay function.

5. The apparatus as defined in claim 1 , wherein execution of the delay tag generates the time delay.

6. The apparatus as defined in claim 1 , wherein execution of the delay tag requests data from a web site associated with generating the time delay.

7. The apparatus as defined in claim 1 , wherein the penalty manager is to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.

8. The apparatus as defined in claim 1 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

9. An apparatus to throttle media access at a server, the method comprising:
means for obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media;
means for determining a source indicator of a media-requesting source, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider; and
means for inserting a time delay including a delay tag in the media response message to the media-requesting source based on the source indicator, the delay tag to cause a delay time period.

10. The apparatus as defined in claim 9 , wherein the media-requesting source is a web crawler.

11. The apparatus as defined in claim 9 , wherein the delay tag includes executable instructions.

12. The apparatus as defined in claim 11 , wherein the executable instructions include a native delay function.

13. The apparatus as defined in claim 9 , wherein executing the delay tag generates the time delay.

14. The apparatus as defined in claim 9 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.

15. The apparatus as defined in claim 9 , wherein inserting the time delay in the media response message comprises selecting a pre-existing copy of the media with a delay tag associated with the time delay.

16. The apparatus as defined in claim 9 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

17. A non-transitory tangible computer readable storage medium comprising instructions that, when executed, cause a server hosting media to at least:
obtain a media request message for media hosted by the server, the media request message requesting access to the media;
determine a source indicator of a media-requesting source, the source indicator indicating whether the media-requesting source is associated with a media provider to provide audience measurement information associated with the media provided by the media provider; and
insert a time delay including a delay tag in the media response message to the media-requesting source based on the characterization, the delay tag to cause a delay time period based on the source indicator.

18. The computer readable storage medium as defined in claim 17 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.

19. The computer readable storage medium as defined in claim 17 , wherein the instructions further cause the server to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.

20. The computer readable storage medium as defined in claim 17 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

Assignees

• Nielsen Co Us Llc

Inventors

• Lawrence Iii Bradley R

Classifications

• H04L47/32

  by discarding or delaying data units, e.g. packets or frames · CPC title

• H04L47/80 Primary

  Actions related to the user profile or the type of traffic · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• G06F17/30864

  Physics · mapped topic