Method and apparatus for accelerated record layout detection

Patent metadata

Field	Value
Publication number	US-10133802-B2
Application number	US-201514694595-A
Country	US
Kind code	B2
Filing date	Apr 23, 2015
Priority date	Oct 23, 2012
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Various methods and apparatuses are described for performing high speed translations of data. In an example embodiment, record layout detection can be performed for data. In another example embodiment, data pivoting prior to field-specific data processing can be performed.

First claim

What is claimed is:

1. An apparatus for processing a record to determine its record layout, the apparatus comprising:
a hardware record layout detector for processing a plurality of records, the records comprising data arranged in a format and exhibiting any of a plurality of different record layouts within the format, wherein the hardware record layout detector is deployed on at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and a cell processor, wherein the hardware record layout detector comprises (1) a plurality of data analysis components arranged in parallel, and (2) logic downstream from the parallel data analysis components; and
a memory;
wherein the parallel data analysis components are configured to analyze the records in parallel, each parallel data analysis component configured to (1) test data in the records against a corresponding condition, and (2) output data indicative of whether the tested data satisfies the corresponding condition, wherein the corresponding conditions for the parallel data analysis components in the aggregate serve as criteria for determining whether the record exhibits any of a plurality of different record layouts, and wherein the corresponding conditions include byte offsets that identify record portions to be processed by different ones of the parallel data analysis components;
wherein the memory is configured to store a configuration table, the configuration table configured to store data representative of the corresponding conditions;
wherein the hardware record layout detector is further configured to define the corresponding conditions for the parallel data analysis components based on the corresponding conditions data in the configuration table; and
wherein the logic is configured to process the output data from the parallel data analysis components to determine, for each of a plurality of the records, whether that record exhibits a record layout from among the plurality of different record layouts;
wherein the hardware record layout detector, in response to a determination that the record exhibits a record layout from among the plurality of different record layouts, is further configured to associate the record with data indicative of the determined record layout; and
wherein the hardware record layout detector is adaptable to detect a new record layout in response to an update to the configuration table that adds corresponding conditions for the new record layout to the configuration table.

2. The apparatus of claim 1 wherein the data analysis components and the logic are deployed on a reconfigurable logic device.

3. The apparatus of claim 2 wherein the records comprise a plurality of records having a fixed field format.

4. The apparatus of claim 2 wherein the records comprise a plurality of records having a mapped field format.

5. The apparatus of claim 4 wherein the mapped field format is a mapped variable field format.

6. The apparatus of claim 2 wherein the records comprise a plurality of records having a delimited data format.

7. The apparatus of claim 2 wherein the reconfigurable logic device is configured to stream the records through the data analysis components and the logic such that the data analysis components and the logic perform their operations on the streaming records.

8. The apparatus of claim 7 further comprising:
a first set of the data analysis components and the logic arranged as a first record layout detection module for records having a first format;
a second set of the data analysis components and the logic arranged as a second record layout detection module for records having a second format; and
a multiplexer configured to selectively deliver records to the first record layout detection module or the second record layout detection module based on the formats for the records.

9. The apparatus of claim 8 wherein the first format comprises a member of the group consisting of a fixed field format, a mapped field format, and a delimited data format; and
wherein the second format comprises a member of the group consisting of a fixed field format, a mapped field format, and a delimited data format such that the second format is not the first format.

10. The apparatus of claim 7 further comprising:
a first set of the data analysis components and the logic arranged as a first record layout detection module for records having a first format;
a second set of the data analysis components and the logic arranged as a second record layout detection module for records having a second format;
a third set of the data analysis components and the logic arranged as a third record layout detection module for records having a third format; and
a multiplexer configured to selectively deliver records to the first record layout detection module, the second record layout detection module, or the third record layout detection module based on the formats for the records.

11. The apparatus of claim 10 wherein the first format comprises a fixed field format, wherein the second format comprises a mapped field format, and wherein the third format comprises a delimited data format.

12. The apparatus of claim 1 wherein each of a plurality of the data analysis components comprises a data range collector component and a data checker component, the data range collector component configured to identify portions of a record to be tested against a condition, and the data checker component configured to test the identified portion of the record against the condition.

13. The apparatus of claim 12 wherein each of a plurality of the data range collector components is further configured to identify the record portion based on a byte offset.

14. The apparatus of claim 13 wherein a plurality of the data range collector components are configured with different byte offsets such that those data range collector components identify different record portions based on the different byte offsets.

15. The apparatus of claim 14 wherein the configuration table is further configured to store the different byte offsets, and wherein the hardware logic detector is further configured to define the byte offsets for the data range collector components based on the different byte offsets in the configuration table.

16. The apparatus of claim 12 wherein each of a plurality of the data range collector components is further configured to identify the record portion based on a field identifier.

17. The apparatus of claim 16 wherein a plurality of the data range collector components are configured with different field identifiers such that those data range collector components identify different record portions based on the different field identifiers.

18. The apparatus of claim 17 wherein the configuration table is further configured to store the different field identifiers, and wherein the hardware logic detector is further configured to define the field identifiers for the data range collector components based on the different field identifiers in the configuration table.

19. The apparatus of claim 12 wherein the data range collector component and the data checker component of each data analysis unit are configured as a predicate evaluation logic pipeline such that the data range collector component and the data checker component of each predicate evaluation logic pipeline are configured to operate simultaneously.

20. The apparatus of claim 1 wherein the data analysis components and the logic are deployed on a field programmable

Assignees

• Ip Reservoir Llc

Inventors

• Lancaster Joseph M
• Sprague Kevin Brian

Classifications

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F21/71

  to assure secure computing or processing of information · CPC title

• G06F17/30598

  Physics · mapped topic

• G06F17/30569 Primary

  Physics · mapped topic