Data storage system ensuring consistency of data received by a producer from a collector via a shared memory

Patent metadata

Field	Value
Publication number	US-12306823-B2
Application number	US-202318100129-A
Country	US
Kind code	B2
Filing date	Jan 23, 2023
Priority date	Jan 23, 2023
Publication date	May 20, 2025
Grant date	May 20, 2025

Abstract

Ensuring consistency of data being received by a producer from a collector via a shared memory includes maintaining a time-of-day (TOD) variable, and, by the producer for new data, (1) first setting the TOD variable to an invalid value, (2) next writing the new data into the shared memory with a timestamp, and (3) next setting the TOD variable to a time value corresponding to the timestamp. The method further includes, by the collector for the new data, (1) reading the TOD variable and timestamp and confirming that the TOD variable has a valid time value equal to the timestamp, (2) reading the new data from the shared memory, and (3) confirming validity of the new data by (a) reading the TOD variable again, and (b) confirming that the TOD variable has the same valid time value as in the reading step (1).

First claim

What is claimed is:

1. A method performed for data being transferred by a producer to a collector via a shared memory at regular intervals, comprising:
maintaining a time-of-day (TOD) variable for storing a time value with a granularity at least as fine as an interval size of the intervals;
by the producer for new data to be written into the shared memory for a given interval, (1) writing the new data into the shared memory with a timestamp for the given interval, and (2) setting the TOD variable to a time value equal to the timestamp; and
by the collector for the new data, (1) reading the TOD variable and timestamp from the shared memory and comparing the TOD variable with the timestamp, (2) in response to the TOD variable being equal to the timestamp, then reading the new data as collected new data from the shared memory, and (3) in response to the TOD variable not being equal to the timestamp, then refraining from reading the new data as collected new data from the shared memory.

2. The method of claim 1 , wherein the new data is first data updated at the regular intervals, and wherein the method further includes, for second data updated by the producer at other than regular intervals:
by the collector for the second data, (1) reading the TOD variable and the timestamp from the shared memory, and checking whether the TOD variable has a time value not older than the timestamp by more than a predefined maximum value, (2) in response to the time value being not older than the timestamp by more than the predefined maximum value, then reading the second data as collected new second data from the shared memory, and (3) in response to the time value being older than the timestamp by more than the predefined maximum value, then refraining from reading the second data as collected new second data from the shared memory.

3. The method of claim 1 , wherein the TOD variable is stored in a data region of the shared memory that also stores the new data being written by the producer and read by the collector, and wherein the shared memory also contains a metadata region in which the producer stores metadata describing structure of the data region, the metadata being read and used by the collector for correctly accessing the new data.

4. The method of claim 3 , wherein the metadata includes an update type and an update interval, the update type indicating that the new data is updated at the regular intervals, the update interval identifying the interval size of the regular intervals, the update type and update interval being used by the collector to control timing of accessing the new data to obtain the new data as the new data is updated by the producer.

5. The method of claim 4 , wherein the update type and update interval are included in a general area of the metadata, and the metadata further includes a plurality of field definition areas each storing a respective field definition describing structure and content of a respective field of each of a plurality of records of the new data, each field definition being used by the collector to locate and interpret a respective field in each of the records of the new data.

6. The method of claim 5 , wherein the field definitions each include at least a field name, field location, and field size for the respective field, the field name identifying contents of the field, the field location identifying a relative location of the field in each of the records, the field size identifying a size of the field.

7. The method of claim 5 , wherein the general area further stores a layout type identifying a high-level structure of the data region, the layout type being selected from an array type and a map type, the array type having the records arranged in a single area with no internal indirection, the map type having the records arranged across multiple areas with at least one indirection from one area to another.

8. The method of claim 3 , wherein the new data is contained in one record of a multi-record data collection being received by the producer from the collector via the shared memory, each record including a respective data generation identifier stored in the data region and a respective metadata generation identifier stored in the metadata region, and wherein the producer and collector operate according to a protocol including:
by the producer:
when initially creating a new record in the data region and when reusing a record of the data region, (i) generating a new data generation identifier and writing the new data generation identifier into the record, (ii) initializing other record fields, (iii) initializing corresponding metadata in the metadata region, and (iv) setting a metadata generation identifier to equal the data generation identifier; and
when removing a record from the data region, changing the data generation identifier to an invalid value, and changing the metadata generation identifier to the invalid value; and

by the collector, for collecting the new data:
copying the entirety of the data collection from the shared memory to private memory of the collector; and
scanning and processing the records of the data by, for each record:
(i) skipping further processing of a record when the metadata generation identifier is invalid and when the metadata generation identifier is not equal to the data generation identifier; and
(ii) collecting record data for use when the metadata generation identifier is valid and equal to the data generation identifier.

9. The method of claim 8 , wherein the collector makes the record data available for use by data users by storing the record data into one of a file and a database.

10. The method of claim 1 , wherein the producer registers with the collector by storing registration data in a collector configuration readable by the collector to identify the producer and begin collecting the data in subsequent operation.

11. A data storage system, comprising:
interface circuitry configured to receive storage commands from remote storage users;
storage devices configured to store user data; and
processing circuitry configured to execute data storage operations using the storage devices in response to the storage commands, the processing circuitry including a subsystem configured to operate, for data being transferred by a producer to a collector via a shared memory at regular intervals, to:
maintain a time-of-day (TOD) variable for storing a time value with a granularity at least as fine as an interval size of the intervals;
by the producer for new data to be written into the shared memory for a given interval, (1) write the new data into the shared memory with a timestamp for the given interval, and (2) set the TOD variable to a time value corresponding to the timestamp; and
by the collector for the new data, (1) read the TOD variable and timestamp from the shared memory and compare the TOD variable with the timestamp, (2) in response to the TOD variable being equal to the timestamp, read the new data as collected new data from the shared memory, and (3) in response to the TOD variable not being equal to the timestamp, then refrain from reading the new data as collected new data from the shared memory.

12. The data storage system of claim 11 , wherein the new data is first data updated at the regular intervals, and wherein the subsystem is further configured, for second data updated by the producer at other than regular intervals, to:
by the collector for the second data, (1) read the TOD variable and the timestamp from the shared memory, and check whether the TOD variable has a time value not older than the timestamp by more than a predefined maximum value, (2) in response to the time value bei

Assignees

• Dell Products Lp

Inventors

• Dong Dazhi
• Harduf Yuval
• Ma Venoy Qiang
• Chen Xing

Classifications

• G06F16/215

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title

• G06F16/2365 Primary

  Ensuring data consistency and integrity · CPC title