Storage medium and compression for object stores

Patent metadata

Field	Value
Publication number	US-11943294-B1
Application number	US-202017039938-A
Country	US
Kind code	B1
Filing date	Sep 30, 2020
Priority date	Sep 30, 2020
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

An object compression system of a storage service may provide a user-friendly and cost-efficient solution to manage stored objects for users of the storage service. The object compression system may automatically monitor individual objects in an object-based storage system of the storage service. Based on one or more characteristics and additional features of the objects, the object compression system may generate respective determinations whether to compress the individual objects. Responsive to determinations to compress the objects, the object compression system may determine compression algorithms for individual objects. The object compression system may compress the objects according to respective compression algorithms to create compressed files. The object compression system may store the compressed files in a tier different from the objects and delete the objects from the storage service.

First claim

What is claimed is:

1. A system, comprising:
a plurality of computing devices, respectively comprising at least one processor and a memory, the plurality of computing devices configured to implement an object-based data storage system of a cloud-based provider network that stores a plurality of objects, wherein the object-based data storage system is configured to:
monitor access to the plurality of objects to determine one or more characteristics of a first object of the plurality of objects stored in the object-based data storage system of the cloud-based provider network;
generate a compression decision for the first data object using a machine learning model, wherein to generate the compression decision, the machine learning model:
accepts as input the one or more determined characteristics of the first data object to incorporate a prediction of future access to the first object stored in the object-based data storage system of the cloud-based provider network as a basis for the compression decision; and
wherein the compression decision generated by the machine learning model includes both an indication to compress the first data object and a compression algorithm to be performed on the first data object;

generate a compressed version of the first object according to the indication in the compression decision to perform compression on the first data object using the compression algorithm in the compression decision generated by the machine learning model; and
select a different location to store the compressed version of the first object that provides different access performance than a current location of the first object.

2. The system of claim 1 , wherein the object-based storage system is further configured to determine a type of the first object and wherein the machine learning model uses as further input the type of the first object to generate the compression decision.

3. The system of claim 1 , wherein the object-based storage system is further configured to:
decompress the compressed version of the first object to validate restoration of the first object from the compressed version, before selecting the different location to store the compressed version of the first object.

4. The system of claim 1 , wherein the object-based storage system is further configured to delete the first object after storing the compressed version of the first object.

5. The system of claim 1 , wherein the object-based data storage system is implemented as part of a storage service of the cloud-based provider network and wherein the first object is identified for automated compression management via an interface for the storage service.

6. A method, comprising:
monitoring access to a first object stored in a first type of data store of a cloud-based provider network to determine one or more characteristics of the first object;
generating a compression decision for the first object using a machine learning model, wherein in generating the compression decision, the machine learning model:
accepts as input the one or more determined characteristics of the first object to incorporate a prediction of future access to the first object stored in the data store of the cloud-based provider network as a basis for the compression decision; and
wherein the compression decision generated by the machine learning model includes both an indication to compress the first object and a compression algorithm to be performed on the first object;

generating a compressed version of the first object according to the indication in the compression decision to perform compression on the first object using the compression algorithm in the compression decision generated by the machine learning model; and
storing the compressed version of the first object in one data store determined from a plurality of possible data stores, wherein the plurality of possible data stores includes the first type of data store.

7. The method of claim 6 ,
wherein the method further comprises determining a type of the first object stored in the first type of data store; and
wherein the machine learning model accepts as further input the type of the first data object to generate the compression decision.

8. The method of claim 6 , wherein one of the one or more characteristics determined from the monitoring is a type of an application accessing the first object stored in the first type of data store.

9. The method of claim 6 , wherein the one data store determined to store the compressed version of the first object is a second type of data store, and wherein the first type of data store and the second type of data store provide different respective access performance.

10. The method of claim 6 , further comprising deleting the first object from the first type of data store.

11. The method of claim 6 , wherein generating the compression decision is performed after expiration of a period of time specified for the first object via an interface.

12. The method of claim 6 , further comprising:
making a determination to compress a second object stored in the first type of data store based, at least in part, on the compression decision made for the first object;
generating a compressed version of the second object; and
storing the respective compressed version of the second object in the one data store.

13. The method of claim 6 , further comprising:
validating restoration of the first object from the compressed version, before selecting the different location to store the compressed version of the first object.

14. The method of claim 6 , further comprising:
receiving one or more compression tolerance characteristics via an interface, wherein the one or more compression tolerance characteristics are used as further input to the machine learning model to generate the compression decision for the first object.

15. The method of claim 6 , wherein making the determination to compress the first object comprises determining that a confidence score for the prediction of future access to the first object is above a threshold.

16. One or more non-transitory computer readable media storing program instructions that when executed on or across one or more computing devices, cause the one or more computing devices to implement:
monitoring access to a first object in a first type of data store of a cloud-based provider network to determine one or more characteristics of the first object, wherein the first object is identified for automated compression via an interface; and
generating a compression decision for the first object using a machine learning model, wherein in generating the compression decision, the machine learning model:
accepts as input the one or more determined characteristics of the first object to incorporate a prediction of future access to the first object stored in the data store of the cloud-based provider network as a basis for the compression decision; and
wherein the compression decision generated by the machine learning model includes both an indication to compress the first object and a compression algorithm to be performed on the first object;

compressing the first object according to the indication in the compression decision to perform compression on the first object using the compression algorithm in the compression decision generated by the machine learning model; and
storing the compressed first object to a second type of data store different from the first type of data store storing the first object.

17. The one or more non-transitory computer readable media of claim 16 ,
wherein the one or m

Assignees

• Amazon Tech Inc

Inventors

• Bartenstein Christoph
• Marshall Brad E
• Warfield Andrew Kent

Classifications

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N20/10

  using kernel methods, e.g. support vector machines [SVM] · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0475

  Generative networks · CPC title