Method and apparatus for accelerated format translation of data in a delimited data format

Patent metadata

Field	Value
Publication number	US-9633093-B2
Application number	US-201314060313-A
Country	US
Kind code	B2
Filing date	Oct 22, 2013
Priority date	Oct 23, 2012
Publication date	Apr 25, 2017
Grant date	Apr 25, 2017

Abstract

Various methods and apparatuses are described for performing high speed format translations of incoming data, where the incoming data is arranged in a delimited data format. As an example, the data in the delimited data format can be translated to a fixed field format using pipelined operations. A reconfigurable logic device can be used in exemplary embodiments as a platform for the format translation.

First claim

What is claimed is:

1. A method comprising:
receiving data in a delimited data format, wherein the received data in the delimited data format comprises (1) a plurality of data characters arranged in a plurality of fields, (2) a plurality of shield characters, and (3) a plurality of field delimiter characters, the field delimiter characters defining a plurality of boundaries between the fields;
converting the received data to a fixed field format, wherein the converting step comprises the reconfigurable logic device (1) distinguishing between field delimiter characters and data characters in the received data based on the shield characters, (2) identifying the fields in the received data based on the field delimiter characters, and (3) arranging the data characters sharing the same identified fields into fixed-size fields such that the converted data comprises the data characters in the fixed-size fields stripped of the field delimiter characters and the shield characters;
performing a plurality of processing operations on the converted data to generate processed data in the fixed field format;
loading the processed data into a database; and
wherein the converting step is performed by a reconfigurable logic device.

2. The method of claim 1 wherein the converted data comprises a plurality of data fields, each data field having a known field length, wherein the processing operations comprise a plurality of field-specific data processing operations, and wherein the performing step comprises targeting a specific field of the converted data for a field-specific processing operation without analyzing the data content of the data fields.

3. The method of claim 1 wherein the data processing operations comprise data quality checking operations as part of an extract, transfer, load (ETL) procedure.

4. The method of claim 1 wherein the at least one of the processing operations is performed by software executed by a processor.

5. The method of claim 1 wherein the converting step comprises converting a plurality of characters of the received data to the fixed field format per clock cycle.

6. The method of claim 1 wherein the at least one of the processing operations is performed by a reconfigurable logic device.

7. A method comprising:
a reconfigurable logic device receiving an incoming stream comprising a plurality of bytes arranged in a delimited data format, the incoming byte stream being representative of data arranged in a plurality of fields, wherein the received byte stream comprises a plurality of data characters, a plurality of field delimiter characters, and a plurality of shield characters, the field delimiter characters defining a plurality of boundaries between the fields;
the reconfigurable logic device processing the received byte stream, wherein the processing step comprises:
the reconfigurable logic device distinguishing between field delimiter characters and data characters in the received byte stream based on the shield characters to identify the field delimiter characters that are present in the received byte stream; and
the reconfigurable logic device identifying the fields in the received byte stream based on the identified field delimiter characters; and

the reconfigurable logic device translating the received byte stream to an outgoing byte stream arranged in a fixed field format based on the identified field delimiter characters, the outgoing byte stream comprising a plurality of the data characters of the received byte stream arranged in a plurality of fixed-size fields, and wherein the translating step comprises the reconfigurable logic device arranging the data characters sharing the same identified field into the fixed-size fields such that the outgoing byte stream comprises the data characters in the fixed-size fields stripped of the field delimiter characters and the shield characters.

8. The method of claim 7
wherein the processing step further comprises the reconfigurable logic device identifying the shield characters that are present in the received byte stream.

9. The method of claim 8 wherein the translating step comprises the reconfigurable logic device removing the identified field delimiter characters from the outgoing byte stream.

10. The method of claim 9 wherein the translating step further comprises the reconfigurable logic device removing the identified shield characters from the outgoing byte stream.

11. The method of claim 8 further comprising the reconfigurable logic device converting the received byte stream to an internal format tagged with associated control data that identifies the boundaries between the fields.

12. The method of claim 11 wherein the converting step further comprises the reconfigurable logic device generating a shield character mask associated with the received byte stream to identify the bytes in the received byte stream that are eligible for consideration as to whether they contain a field delimiter character.

13. The method of claim 12 wherein the converting step further comprises the reconfigurable logic device processing the bytes of the received byte stream and the generated shield character mask to generate field delimiter flag data associated with the received byte stream, the field delimiter flag data being indicative of whether an associated byte corresponds to a field delimiter character.

14. The method of claim 13 wherein the incoming byte stream is further representative of a plurality of records, at least one of the records comprising at least one of the fields, the incoming byte stream further comprising a plurality of record delimiter characters, the record delimiter characters defining a plurality of boundaries between the records, and wherein the converting step further comprises the reconfigurable logic device processing the bytes of the received byte stream and the generated shield character mask to generate record delimiter flag data associated with the received byte stream, the record delimiter flag data being indicative of whether an associated byte corresponds to a record delimiter character.

15. The method of claim 14 wherein the converting step further comprises the reconfigurable logic device identifying any empty fields that exist within the received byte stream based on the field delimiter flag data and the record delimiter flag data.

16. The method of claim 15 wherein the converting step further comprises the reconfigurable logic device removing the field delimiter characters and the record delimiter characters from the internally formatted byte stream based on the field delimiter flag data and the record delimiter flag data.

17. The method of claim 16 wherein the converting step further comprises the reconfigurable logic device generating control data associated with the internally formatted byte stream, the control data comprising (1) a start of field flag, (2) an end of field flag, (3) a start of record flag, (4) an end of record flag, and (5) a field identifier.

18. The method of claim 11 wherein the shield character identifying step further comprises the reconfigurable logic device performing a shield character removal operation on the bytes of the received byte stream.

19. The method of claim 18 wherein the shield character removal performing step comprises the reconfigurable logic device (1) distinguishing between the data characters that match the shield character and the shield characters, and (2) removing the identified shield characters.

20. The method of claim 11 further comprising the reconfigurable logic

Assignees

• Ip Reservoir Llc

Inventors

• Henrichs Michael John
• Lancaster Joseph M
• Chamberlain Roger Dean
• White Jason R
• Sprague Kevin Brian
• Tidwell Terry

Classifications

• G16H10/60

  for patient-specific data, e.g. for electronic patient records · CPC title

• G06F16/283

  Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP · CPC title

• G06F16/254 Primary

  Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses · CPC title

• G06F16/258

  Data format conversion from or to a database · CPC title

• G06F17/30592

  Physics · mapped topic