Method and apparatus to throttle media access by web crawlers

Patent metadata

Field	Value
Publication number	US-9887933-B2
Application number	US-201414530659-A
Country	US
Kind code	B2
Filing date	Oct 31, 2014
Priority date	Oct 31, 2014
Publication date	Feb 6, 2018
Grant date	Feb 6, 2018

Abstract

Methods, apparatus, systems and articles of manufacture are disclosed to throttle resource access by web crawlers. An example method disclosed herein includes obtaining, at a server, a media request message for media hosted by the server, the media request message requesting access to the media, characterizing a media-requesting source associated with the media request message, and inserting a time delay in a media response message to the media-requesting source based on the characterization.

First claim

What is claimed is:

1. A method to throttle media access at a server, the method comprising:
obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media;
extracting source-identifying information from the media request message to identify a media-requesting source associated with the media request message;
matching the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information to determine a partner affiliation status of the media-requesting source, the partner affiliation status indicating whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server;
and
inserting a time delay including a delay tag in the media response message to the media-requesting source based on the partner affiliation status, the delay tag to cause a delay time period.

2. The method as defined in claim 1 , wherein the media-requesting source is a web crawler.

3. The method as defined in claim 1 , wherein the delay tag includes executable instructions.

4. The method as defined in claim 3 , wherein the executable instructions include a native delay function.

5. The method as defined in claim 1 , wherein executing the delay tag generates the time delay.

6. The method as defined in claim 1 , wherein executing the delay tag causes the media-requesting source to request data from a web site associated with generating the time delay.

7. The method as defined in claim 1 , wherein inserting the time delay in the media response message comprises selecting a pre-existing copy of the media with a delay tag associated with the time delay.

8. The method as defined in claim 1 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

9. A system comprising:
a data interface to obtain a media request message for media hosted by a server, the media request message requesting access to the media;
a category handler to determine a media-requesting source associated with the media request message and a partner affiliation status of the media-requesting source, the partner affiliation status indicating whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server, and extract source-identifying information from the media request message, match the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information;
a penalty manager to insert a time delay including a delay tag in the media response message to the media-requesting source based on the partner affiliation status, the delay tag to cause a delay time period based on the source-identifying information at least one of the data interface, category handler, and penalty manager is implemented on a logic circuit.

10. The system as defined in claim 9 , wherein the media-requesting source is a web crawler.

11. The system as defined in claim 9 , wherein the delay tag includes executable instructions.

12. The system as defined in claim 11 , wherein the executable instructions include a native delay function.

13. The system as defined in claim 9 , wherein execution of the delay tag generates the time delay.

14. The system as defined in claim 9 , wherein execution of the delay tag requests data from a web site associated with generating the time delay.

15. The system as defined in claim 9 , wherein the penalty manager is to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.

16. The system as defined in claim 9 , wherein the media request message is either a hypertext transfer protocol GET request or a hypertext transfer protocol POST request.

17. A tangible computer readable storage medium comprising instructions that, when executed, cause a server hosting media to at least:
obtain a media request message for media hosted by the server, the media request message requesting access to the media;
extract source-identifying information from the media request message;
match the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information to determine a partner affiliation status of the media-requesting source, the partner affiliation status indicating whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server; and
insert a time delay including a delay tag in the media response message to the media-requesting source based on the characterization, the delay tag to cause a delay time period based on the source-identifying information.

18. The tangible computer readable storage medium as defined in claim 17 , wherein the instructions further cause the server to insert the time delay in the media response message by selecting a pre-existing copy of the media with a delay tag associated with the time delay.

19. A method to throttle media access at a server, the method comprising:
obtaining, at the server, a media request message for media hosted by the server, the media request message requesting access to the media;
extracting source-identifying information from the media request message to identify a media-requesting source associated with the media request message;
matching the source-identifying information with a source category identifier from a data collection facility in a lookup table with the source-identifying information to determine a partner affiliation status of the media-requesting source;
determining a time delay for a media response message based on the partner affiliation status; and
inserting the time delay including a delay tag in the media response message to the media-requesting source based on the partner affiliation status, the delay tag to cause a delay time period,
wherein the partner affiliation status indicates whether the media-requesting source has a business partnership with a media provider to provide audience measurement information associated with the server.

Assignees

• Nielsen Co Us Llc

Inventors

• Lawrence Iii Bradley R

Classifications

• H04L47/80 Primary

  Actions related to the user profile or the type of traffic · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• H04L47/32

  by discarding or delaying data units, e.g. packets or frames · CPC title

• G06F17/30864

  Physics · mapped topic