Selectively evicting data from internal memory during record processing

Patent metadata

Field	Value
Publication number	US-11513704-B1
Application number	US-202117403630-A
Country	US
Kind code	B1
Filing date	Aug 16, 2021
Priority date	Aug 16, 2021
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

A computer-implemented method, according to one embodiment, includes: processing records by, for each of the records: shearing the key associated with the record from the payload data, normalizing the sheared key, and storing the normalized sheared key in a first target area of memory. A determination is made whether a size of the payload data is outside a first predetermined range. In response to determining that the size of the payload data is outside the first predetermined range, the payload data is stored in a second target area of memory, and a data locator is appended to the normalized sheared key. Furthermore, in response to determining that a storage capacity of the memory is outside a second predetermined range, some of the payload data is transferred to external physical storage. Moreover, an external list is integrated with each of the data locators that correspond to the transferred payload data.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
processing records in an input data buffer by performing the following operations on each of the records, wherein each of the records include a key which is appended to payload data in the respective record:
causing the key associated with the record to be sheared from the payload data in the record,
causing the sheared key to be normalized,
causing the normalized sheared key to be stored in a first target area of memory,
determining whether a size of the payload data in the record is outside a first predetermined range,
in response to determining that the size of the payload data in the record is outside the first predetermined range, causing the payload data to be stored in a second target area of memory, and
causing a data locator to be appended to the normalized sheared key in the first target area of memory to form a sheared record, wherein the data locator corresponds to a storage location of the payload data in the second target area of memory;

in response to determining that a storage capacity of the memory is outside a second predetermined range, causing at least some of the payload data to be transferred to external physical storage; and
causing an external list to be integrated with each of the data locators that correspond to the transferred payload data, wherein the external list corresponds to a storage location of the transferred payload data in the external physical storage.

2. The computer-implemented method of claim 1 , comprising:
in response to determining that the size of the payload data in the record is not outside the first predetermined range, causing the normalized sheared key to be combined with the corresponding payload data to form an appended record; and
integrating the appended record to be sorted in the first target area of memory.

3. The computer-implemented method of claim 2 , wherein causing at least some of the payload data to be transferred to external physical storage includes:
selecting payload data and the corresponding normalized sheared key from one or more of the sheared records;
causing the selected payload data and the selected normalized sheared keys to be sent to the external physical storage; and
causing the selected payload data and the selected normalized sheared keys to be stored in the external physical storage.

4. The computer-implemented method of claim 3 , wherein causing the selected payload data and the selected normalized sheared keys to be sent to the external physical storage includes:
causing the selected payload data to be sent to the external physical storage; and
causing the selected normalized sheared keys to be sent the external physical storage in response to the selected payload data being sent to the external physical storage.

5. The computer-implemented method of claim 1 , comprising:
receiving a request for payload data in the external physical storage;
using one or more of the external lists to determine the storage location of the requested payload data in the external physical storage;
sending one or more instructions to retrieve the requested payload data from the storage location in the external physical storage;
causing the requested payload data to be stored in the second target area of memory; and
causing the data locator corresponding to the requested payload data to be updated such that the data locator corresponds to a storage location of the requested payload data in the second target area of memory.

6. The computer-implemented method of claim 5 , wherein updating the data locator corresponding to the requested payload data includes removing the external list from the data locator.

7. The computer-implemented method of claim 1 , wherein causing a data locator to be appended to the normalized sheared key in the first target area of memory to form a sheared record includes:
determining whether the record is a variable length record;
in response to determining that the record is a variable length record, causing an offset value and a length to be implemented in the data locator appended to the normalized sheared key; and
in response to determining that the record is not a variable length record, causing a pointer to be used as the data locator appended to the normalized sheared key.

8. The computer-implemented method of claim 1 , comprising:
causing a sorting operation to be performed on the sheared records in the first target area of memory.

9. The computer-implemented method of claim 1 , comprising:
in response to determining that the size of the payload data in the record is not outside the first predetermined range, causing the normalized sheared key to be combined with the corresponding payload data to form an appended record,
combining the appended record to be stored in the first target area of memory, and
causing a sorting operation to be performed on the appended records in the first target area of memory.

10. A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable and/or executable by a processor to cause the processor to:
process, by the processor, records in an input data buffer by performing the following operations on each of the records, wherein each of the records include a key which is appended to payload data in the respective record:
cause, by the processor, the key associated with the record to be sheared from the payload data in the record,
cause, by the processor, the sheared key to be normalized,
cause, by the processor, the normalized sheared key to be stored in a first target area of memory,
determine, by the processor, whether a size of the payload data in the record is outside a first predetermined range,
in response to determining that the size of the payload data in the record is outside the first predetermined range, cause, by the processor, the payload data to be stored in a second target area of memory, and
cause, by the processor, a data locator to be appended to the normalized sheared key in the first target area of memory to form a sheared record, wherein the data locator corresponds to a storage location of the payload data in the second target area of memory;

in response to determining that a storage capacity of the memory is outside a second predetermined range, cause, by the processor, at least some of the payload data to be transferred to external physical storage; and
cause, by the processor, an external list to be integrated with each of the data locators that correspond to the transferred payload data, wherein the external list corresponds to a storage location of the transferred payload data in the external physical storage.

11. The computer program product of claim 10 , wherein the program instructions are readable and/or executable by the processor to cause the processor to:
in response to determining that the size of the payload data in the record is not outside the first predetermined range, cause, by the processor, the normalized sheared key to be combined with the corresponding payload data to form an appended record; and
integrate, by the processor, the appended record to be sorted in the first target area of memory.

12. The computer program product of claim 11 , wherein causing at least some of the payload data to be transferred to external physical storage includes:
selecting payload data and the corresponding normalized sheared key from one or more of the sheared records;
causing the selected payload data and the selected normalized sheared keys to be sent to the external physical storage; and

Assignees

• IBM

Inventors

• Compton Scott B
• Suarez Jeffrey Richard
• Pardini Matthew Michael Garcia
• Jacobi Christian

Classifications

• G06F11/3037

  where the computing system component is a memory, e.g. virtual memory, cache (accessing, addressing or allocating within memory systems or architectures G06F12/00; checking stores for correct operation G11C29/00) · CPC title

• G06F3/0604

  Improving or facilitating administration, e.g. storage management · CPC title

• G06F3/0659

  Command handling arrangements, e.g. command buffers, queues, command scheduling · CPC title

• G06F3/067

  Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS] · CPC title

• G06F3/0631 Primary

  by allocating resources to storage systems · CPC title