Method and apparatus for record pivoting to accelerate processing of data fields

Patent metadata

Field	Value
Publication number	US-9633097-B2
Application number	US-201514694622-A
Country	US
Kind code	B2
Filing date	Apr 23, 2015
Priority date	Oct 23, 2012
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

Various methods and apparatuses are described for performing high speed translations of data. In an example embodiment, record layout detection can be performed for data. In another example embodiment, data pivoting prior to field-specific data processing can be performed.

First claim

What is claimed is:

1. An apparatus comprising:
a processor;
a memory comprising a first buffer, a second buffer, and a third buffer; and
a member of the group consisting of (1) a graphics processor unit (GPU), (2) a multi-core processor, (3) a cell processor, and (4) a reconfigurable logic device;
wherein the processor is configured to (1) receive a plurality of incoming records, the incoming records comprising a plurality of fields, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the member, and (4) write the identified fields to the second buffer;
wherein the member is configured to (1) read the fields in the second buffer, (2) perform a data processing operation on the read fields to generate a plurality of processed fields, and (3) write the processed fields to the third buffer; and
wherein the processor is further configured to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate a plurality of outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

2. The apparatus of claim 1 wherein the processor is configured to execute a first software module and a second software module;
the first software module comprising a plurality of instructions configured to cause the processor to (1) receive the incoming records, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the member, and (4) write the identified fields to the second buffer;
the second software module comprising a plurality of instructions configured to cause the processor to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate the outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

3. The apparatus of claim 1 wherein the memory further comprises:
an input ring buffer in which the incoming records are stored; and
an output ring buffer in which the outgoing records are stored;
wherein the processor is configured to (1) receive the incoming records from the input ring buffer and (2) write the outgoing records into the output ring buffer.

4. The apparatus of claim 1 wherein the member is configured to perform an encryption operation on the fields read from the second buffer.

5. The apparatus of claim 4 wherein the encryption operation is format preserving encryption (FPE).

6. The apparatus of claim 1 wherein the second buffer comprises a pool of ingress buffers.

7. The apparatus of claim 6 wherein different fields of the incoming records are written to different ingress buffers to support multi-field processing by the member.

8. The apparatus of claim 1 wherein the third buffer comprises a pool of egress buffers.

9. The apparatus of claim 1 wherein the second buffer comprises a pool of buffers that can be selectively allocated as ingress buffers or egress buffers.

10. The apparatus of claim 1 wherein the member comprises the GPU.

11. The apparatus of claim 10 wherein the GPU is configured to perform an encryption operation on the fields read from the second buffer.

12. The apparatus of claim 11 wherein the encryption operation is format preserving encryption (FPE).

13. The apparatus of claim 10 wherein the second buffer comprises a pool of ingress buffers.

14. The apparatus of claim 13 wherein different fields of the incoming records are written to different ingress buffers to support multi-field processing by the GPU.

15. The apparatus of claim 10 wherein the memory further comprises:
an input ring buffer in which the incoming records are stored; and
an output ring buffer in which the outgoing records are stored;
wherein the processor is configured to (1) receive the incoming records from the input ring buffer and (2) write the outgoing records into the output ring buffer.

16. The apparatus of claim 10 wherein the processor is configured to execute a first software module and a second software module;
the first software module comprising a plurality of instructions configured to cause the processor to (1) receive the incoming records, (2) write the incoming records to the first buffer, (3) identify which of the fields of the incoming records are to be processed by the GPU, and (4) write the identified fields to the second buffer;
the second software module comprising a plurality of instructions configured to cause the processor to (1) read the processed fields in the third buffer, (2) read the fields for the incoming records in the first buffer, and (3) merge the processed fields with the fields from the incoming records to generate the outgoing records such that the outgoing records include the processed fields in place of their corresponding fields of the incoming records.

17. The apparatus of claim 10 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type across different streaming records for grouping in the second buffer.

18. The apparatus of claim 17 wherein the GPU is further configured to perform the data processing operation on the fields read from the second buffer as a batch.

19. The apparatus of claim 18 wherein the streaming records include a plurality of social security number fields;
wherein the processor is further configured to pivot the social security number fields from a plurality of different streaming records into the second buffer; and
wherein the GPU is further configured to (1) encrypt the grouped social security number fields in the second buffer and (2) write the encrypted social security number fields into the third buffer.

20. The apparatus of claim 1 wherein the member comprises the multi-core processor.

21. The apparatus of claim 20 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type across different streaming records for grouping in the second buffer.

22. The apparatus of claim 1 wherein the member comprises the cell processor.

23. The apparatus of claim 22 wherein the processor is further configured to receive the incoming records as streaming records embodied in a data stream, the data stream comprising data arranged sequentially in a plurality of the fields, wherein each of a plurality of the fields of the streaming records has an associated field type, and wherein the processor is further configured to pivot a plurality of the identified fields of the same field type acro

Assignees

• Ip Reservoir Llc

Inventors

• Tidwell Terry
• St John Alex
• Sewell Daniel

Classifications

• G06F16/258 Primary

  Data format conversion from or to a database · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F21/71

  to assure secure computing or processing of information · CPC title

• G06F17/30569 Primary

  Physics · mapped topic

• G06F17/30598

  Physics · mapped topic