Dynamic memory management techniques

Patent metadata

Field	Value
Publication number	US-2018300067-A1
Application number	US-201815945637-A
Country	US
Kind code	A1
Filing date	Apr 4, 2018
Priority date	Apr 12, 2017
Publication date	Oct 18, 2018
Grant date	—

Abstract

Techniques for dynamically changing the amount of memory that is allocated to and used by processes (e.g., sorter processes) executed concurrently by a data processing system in a manner that is adaptive to the overall memory usage of the system. The ability to dynamically change the amount of memory that can be allocated to and used by individual processes increases and optimizes the overall memory utilization of the data processing system, which in turn results in the operations (e.g., sorting) performed by the processes to be performed in a more efficient manner. A memory manager is provided that is responsible for periodically collecting memory related information for the data processing system and for the processes (e.g., sorter processes) executed by the data processing system, and based on the collected information, for periodically and dynamically causing the memory used by the processes to be changed during the lifetime of the processes.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
executing, by a data processing system, a plurality of sorters, each sorter in the plurality of sorters configured to perform a sort operation, the data processing system comprising one or more processors and associated memory; executing, by the data processing system, a memory manager; determining, by the memory manager, a current memory usage for each sorter in the plurality of sorters; based upon the current memory usage determined for each sorter in the plurality of sorters, determining, by the memory manager, a first sorter total value indicative of the total memory usage of the plurality of sorters; determining that the first sorter total value exceeds a first threshold value; responsive to determining that the first sorter total value exceeds the first threshold, calculating, by the memory manager, a first per-sorter limit value based upon the first sorter total value and the first threshold value; and for each sorter in the plurality of sorters whose current memory usage exceeds the first per-sorter limit value, releasing an amount of memory used by the sorter, wherein an amount of memory used by the sorter after the releasing is equal to or less than the first per-sorter limit.

2 . The method of claim 1 further comprising:
for each sorter in the plurality of sorters whose memory usage exceeds the first per-sorter limit value:
providing the memory released from the sorter to the memory manager; and
adding, by the memory manager, the memory released from the sorter to a list of buffers managed by the memory manager.

3 . The method of claim 2 , wherein the releasing comprises releasing an amount of memory used by a first sorter from the plurality of sorters, wherein an amount of memory used by the first sorter after the releasing is equal to or less than the first per-sorter limit;
wherein the providing comprises providing the memory released from the first sorter to the memory manager; and wherein the adding comprises adding the memory released from the first sorter to the list of buffers managed by the memory manager; and wherein the method further comprises:
executing, by the data processing system, a memory allocator, the memory allocator configured to allocate memory for the plurality of sorters, wherein the memory freed by the first sorter is memory allocated to the first sorter by the memory allocator;
receiving, by the memory manager, a request for memory from a second sorter from the plurality of sorters; and
responsive to receiving the request, providing to the second sorter by the memory manager from the list of buffers, a portion of memory released from the first sorter.

4 . The method of claim 2 , further comprising:
executing, by the data processing system, a memory allocator, the memory allocator configured to allocate memory for the plurality of sorters, wherein the memory released by the first sorter is memory allocated to the first sorter by the memory allocator; upon determining that the first sorter total value exceeds the first threshold value releasing, by the memory manager, the list of buffers to the memory allocator.

5 . The method of claim 1 , further comprising:
determining, by the memory manager, a first parameter configured for the data processing system, the first parameter identifying a maximum amount of memory that can be used on the data processing system for sorting operation; determining, by the memory manager, a second parameter configured for the data processing system, the second parameter identifying a memory usage threshold when corrective actions to adjust memory usage of the plurality of sorters is to be performed; and determining the first threshold value based upon the first parameter and the second parameter.

6 . The method of claim 1 , wherein calculating the first per-sorter limit value comprises:
determining, by the memory manager, based on the first sorter total value and the first threshold value, a target total memory value indicative of an amount of memory to release; determining, by the memory manager, a subset of sorters from the plurality of sorters, wherein each sorter from the plurality of sorters whose current memory usage exceeds the target total memory value is included in the subset of sorters; determining, by the memory manager, the first per-sorter limit value based on the sorters in the subset of sorters and their current memory usages, and the target total memory value.

7 . The method of claim 1 , further comprising:
after expiration of a period of time after calculation of the first per-sorter limit value:
determining, by the memory manager, an updated current memory usage for each sorter in the plurality of sorters;
based upon the updated current memory usage determined for each sorter in the plurality of sorters, determining, by the memory manager, a second sorter total value indicative of the total memory usage of the plurality of sorters;
determining that the second sorter total value exceeds the first threshold value;
responsive to determining that the second sorter total value exceeds the first threshold, calculating, by the memory manager, a second per-sorter limit value based upon the second sorter total value and the first threshold value, wherein the second per-sorter limit value is different from the first per-sorter limit value; and
for each sorter in the plurality of sorters whose updated memory usage exceeds the second per-sorter limit value, releasing an amount of memory used by the sorter, wherein an amount of memory used by the sorter after the releasing is equal to or less than the second per-sorter limit.

8 . The method of claim 1 , wherein determining, by the memory manager, the current memory usage for each sorter in the plurality of sorters comprises:
for each sorter in the plurality of sorters, the sorter communicating its current memory usage information to the memory manager.

9 . The method of claim 1 , wherein the releasing the amount of memory used by the sorter is performed by the sorter.

10 . A non-transitory computer-readable medium containing instructions that, when executed by a processor, causes the processor to:
execute a plurality of sorters, each sorter in the plurality of sorters configured to perform a sort operation, the data processing system comprising one or more processors and associated memory; execute a memory manager; determine, by the memory manager, a current memory usage for each sorter in the plurality of sorters; based upon the current memory usage determined for each sorter in the plurality of sorters, determine, by the memory manager, a first sorter total value indicative of the total memory usage of the plurality of sorters; determine that the first sorter total value exceeds a first threshold value; responsive to determining that the first sorter total value exceeds the first threshold, calculate, by the memory manager, a first per-sorter limit value based upon the first sorter total value and the first threshold value; and for each sorter in the plurality of sorters whose current memory usage exceeds the first per-sorter limit value, release an amount of memory used by the sorter, wherein an amount of memory used by the sorter after the releasing is equal to or less than the first per-sorter limit.

11 . The non-transitory computer-readable medium of claim 10 , wherein the instructions, when executed by the processor, further cause the processor to:
for each sorter in the plurality of sorters whose memory usage exceeds the first per-sorter limit value:
provide the memory released from the sorter to the memory manager; and
add,

Assignees

• Oracle Int Corp

Inventors

• Mittal Ashish
• Eng Kenneth Khiaw Hong

Classifications

• G06F11/3034

  where the computing system component is a storage system, e.g. DASD based or network based (digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097) · CPC title

• G06F3/0631

  by allocating resources to storage systems · CPC title

• G06F12/0223

  User address space allocation, e.g. contiguous or non contiguous base addressing · CPC title

• G06F2212/62

  Details of cache specific to multiprocessor cache arrangements · CPC title

• G06F9/5022

  Mechanisms to release resources · CPC title