Data ingest optimization

Patent metadata

Field	Value
Publication number	US-9589065-B2
Application number	US-201213604096-A
Country	US
Kind code	B2
Filing date	Sep 5, 2012
Priority date	Jan 28, 2011
Publication date	Mar 7, 2017
Grant date	Mar 7, 2017

Abstract

Methods and systems for optimizing the retrieval of data from multiple sources are described. A slot map including slots for the storage of data elements can be obtained. The data elements associated with the slots can be prioritized by weighting values with costs of retrieving the data elements from respective data sources. Each value can be associated with a different data element and can indicate a respective degree of importance of the associated data element. Further, the systems and methods can direct the retrieval of data elements from the respective data sources in an order in accordance with the priority of the data elements to optimize the quality of data obtainable within a critical time constraint. In addition, the retrieved data elements can be stored in corresponding slots on a storage medium.

First claim

What is claimed is:

1. A computer readable storage medium comprising a non-transitory computer readable program code, wherein the computer readable program code when executed on a computer causes the computer to:
obtain a slot map including slots for the storage of data segments in a persistent storage device, the slot map including quality tag values associated with each of the data segments applied to each slot in the slot map;
prioritize the data segments associated with the slots by weighting the quality tag values, each of which is associated with a different slot and data segment and indicates a respective degree of importance of the associated data segment, with costs of retrieving the data segments from respective data sources and probabilities of successfully retrieving valid data segments from each of the respective data sources at one or more particular future times, and output a priority queue of the data segments;
populate the slot map with retrieved data segments and output the slot map; and
direct a retrieval of the data segments from the respective data sources in an order in accordance with a determined priority of the data segments to optimize a quality of data obtainable within a critical time constraint.

2. The computer readable storage medium of claim 1 , wherein the data segments provide material for analysis of a subject and wherein each value indicates a respective degree of importance of a corresponding data segment in the analysis.

3. The computer readable storage medium of claim 1 , wherein each value is based upon an expectation of success of retrieving the data segment associated with the value from a corresponding data source.

4. The computer readable storage medium of claim 1 , wherein each value is based upon an expected resource expenditure of retrieving the data segment associated with the value from a corresponding data source.

5. The computer readable storage medium of claim 1 , wherein the retrieval is constrained by at least one of a resource budget or a hard-stop end time.

6. The computer readable storage medium of claim 1 , wherein the retrieval comprises adding additional slots to the slot map and repeating the prioritize step for the additional slots.

7. A system for optimizing the retrieval of data from multiple sources comprising:
a slot map generator configured to generate a slot map including slots for the storage of data segments in a persistent storage device, the slot map further including quality tag values associated with each of the data segments applied to each slot in the slot map;
a priority module configured to prioritize data segments associated with the slots by weighting the quality tag values, each of which is associated with a different slot and data segment and indicates a respective degree of importance of the associated data segment, with costs of retrieving the data segments from respective data sources and probabilities of successfully retrieving valid data segments from each of the respective data sources at one or more particular future times; and
a processor configured to direct a retrieval of the data segments from the respective data sources in an order in accordance with a determined priority of the data segments to optimize a quality of data obtainable within a critical resource constraint, the processor being further configured to output a priority queue of the data segments, populate the slot map with retrieved data segments, and output the slot map.

8. The system of claim 7 , wherein the data segments provide material for analysis of a subject and wherein each value indicates a respective degree of importance of a corresponding data segment in the analysis.

9. The system of claim 7 , wherein the priority module is further configured to base each value upon an expectation of success of retrieving the data segment associated with the value from a corresponding data source.

10. The system of claim 7 , wherein the priority module is further configured to base each value upon an expected resource expenditure of retrieving the data segment associated with the value from a corresponding data source.

11. The system of claim 7 , wherein the critical resource constraint is at least one of a resource budget or a critical time constraint.

12. The system of claim 7 , wherein the processor is further configured to add additional slots to the slot map and to repeat the prioritizing for the additional slots.

Assignees

• Bhagwan Varun
• Grandison Tyrone W A
• Gruhl Daniel F
• IBM

Inventors

• Bhagwan Varun
• Grandison Tyrone W A
• Gruhl Daniel F

Classifications

• G06F16/951 Primary

  Indexing; Web crawling techniques · CPC title

• G06F16/90339 Primary

  by using parallel associative memories or content-addressable memories · CPC title

• G06F17/30899 Primary

  Physics · mapped topic

• G06F16/957 Primary

  Browsing optimisation, e.g. caching or content distillation · CPC title