Method and Apparatus for Accelerated Record Layout Detection

Patent metadata

Field	Value
Publication number	US-2019079984-A1
Application number	US-201816189659-A
Country	US
Kind code	A1
Filing date	Nov 13, 2018
Priority date	Apr 23, 2014
Publication date	Mar 14, 2019
Grant date	—

Abstract

Various methods and apparatuses are described for performing high speed translations of data. In an example embodiment, record layout detection can be performed for data. In another example embodiment, data pivoting prior to field-specific data processing can be performed.

First claim

What is claimed is:

1 . An apparatus for processing a record to determine its record layout, the apparatus comprising:
a hardware record layout detector for processing a plurality of records, the records comprising data arranged in a format and exhibiting any of a plurality of different record layouts within the format, wherein the hardware record layout detector is deployed on at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and/or a cell processor, wherein the hardware record layout detector comprises (1) a plurality of data analysis components arranged in parallel, and (2) logic downstream from the data analysis components; wherein the parallel data analysis components are configured to analyze the records in parallel, each parallel data analysis component configured to (1) test data in the records against a corresponding condition, and (2) output data indicative of whether the tested data satisfies the corresponding condition, wherein the corresponding conditions for the parallel data analysis components in the aggregate serve as criteria for determining whether the record exhibits at least one record layout; and wherein the logic is configured to process the output data from the parallel data analysis components to determine whether the record exhibits the at least one record layout.

2 . The apparatus of claim 1 wherein at least one of the parallel data analysis components comprises a data range collector component and a data checker component, the data range collector component configured to identify portions of a record to be tested against a condition, and the data checker component configured to test the identified portion of the record against the condition.

3 . The apparatus of claim 2 wherein the data range collector component is further configured to identify the record portion based on a byte offset.

4 . The apparatus of claim 2 wherein the data range collector component is further configured to identify the record portion based on a field identifier.

5 . The apparatus of claim 2 wherein the data range collector component and the data checker component of the at least one parallel data analysis component are configured as a predicate evaluation logic pipeline such that the data range collector component and the data checker component of the predicate evaluation logic pipeline are configured to operate simultaneously.

6 . The apparatus of claim 1 wherein each of a plurality of the parallel data analysis components comprises a data range collector component and a data checker component, the data range collector component configured to identify portions of a record to be tested against a condition, and the data checker component configured to test the identified portion of the record against the condition.

7 . The apparatus of claim 6 wherein a plurality of the data range collector components are further configured to identify their respective record portion based on different byte offsets.

8 . The apparatus of claim 6 wherein a plurality of the data range collector components are further configured to identify their respective record portion based on different field identifiers.

9 . The apparatus of claim 1 wherein the record comprises a record having a fixed field format.

10 . The apparatus of claim 1 wherein the record comprises a record having a mapped field format.

11 . The apparatus of claim 10 wherein the mapped field format is a mapped variable field format.

12 . The apparatus of claim 1 wherein the record comprises a record having a delimited data format.

13 . The apparatus of claim 1 wherein:
a first set of the parallel data analysis components and the logic are arranged as a first record layout detector for records having a first format;
a second set of the parallel data analysis components and the logic are arranged as a second record layout detector for records having a second format; and
wherein the hardware record layout detector further comprises a multiplexer configured to selectively deliver records to the first record layout detector or the second record layout detector based on the formats for the records.

14 . The apparatus of claim 13 wherein the first format comprises a member of the group consisting of a fixed field format, a mapped field format, and a delimited data format; and
wherein the second format comprises a member of the group consisting of a fixed field format, a mapped field format, and a delimited data format such that the second format is not the first format.

15 . The apparatus of claim 1 wherein:
a first set of the parallel data analysis components and the logic are arranged as a first record layout detector for records having a first format;
a second set of the parallel data analysis components and the logic are arranged as a second record layout detector for records having a second format;
a third set of the parallel data analysis components and the logic are arranged as a third record layout detector for records having a third format; and
wherein the hardware record layout detector further comprises a multiplexer configured to selectively deliver records to the first record layout detector, the second record layout detector, or the third record layout detector based on the formats for the records.

16 . The apparatus of claim 15 wherein the first format a fixed field format, wherein the second format comprises a mapped field format, and wherein the third format comprises a delimited data format.

17 . The apparatus of claim 1 wherein the parallel data analysis components and the logic are deployed on the reconfigurable logic device.

18 . The apparatus of claim 17 wherein the parallel data analysis components and the logic are deployed on a field programmable gate array (FPGA).

19 . The apparatus of claim 1 wherein the parallel data analysis components and the logic are deployed on the GPU.

20 . The apparatus of claim 1 wherein the parallel data analysis components and the logic are deployed on the multi-core processor.

21 . The apparatus of claim 1 wherein the parallel data analysis components and the logic are deployed on the cell processor.

22 . A method comprising:
streaming a plurality of records through a hardware record layout detector, the records comprising data arranged in a format and exhibiting any of a plurality of different record layouts within the format, wherein the hardware record layout detector is deployed on at least one of a reconfigurable logic device, a graphics processing unit (GPU), a multi-core processor, and/or a cell processor, and wherein the hardware record layout detector comprises (1) a plurality of data analysis components arranged in parallel, and (2) logic downstream from the parallel data analysis components; the parallel data analysis components analyzing the streaming records in parallel, wherein the analyzing step comprises each parallel data analysis component (1) testing data in the streaming records against a corresponding condition, and (2) outputting data indicative of whether the tested data satisfies the corresponding condition, wherein the corresponding conditions for the parallel data analysis components in the aggregate serve as criteria for determining whether the records exhibit at least one record layout; and the logic processing the output data from the parallel data analysis components to determine whether the any of the streaming records exhibit the a

Assignees

• Ip Reservoir Llc

Inventors

• Lancaster Joseph M
• Sprague Kevin Brian

Classifications

• G06F21/71

  to assure secure computing or processing of information · CPC title

• G06F17/30598

  Physics · mapped topic

• G06F17/30569 Primary

  Physics · mapped topic

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title

• G06F16/285

  Clustering or classification · CPC title