Message compression in scalable messaging system

Patent metadata

Field	Value
Publication number	US-10404647-B2
Application number	US-201615175588-A
Country	US
Kind code	B2
Filing date	Jun 7, 2016
Priority date	Jun 7, 2016
Publication date	Sep 3, 2019
Grant date	Sep 3, 2019

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving from a plurality of publisher clients a plurality of messages, each message being for a particular channel of a plurality of distinct channels wherein each channel comprises an ordered plurality of messages, encoding each message based on a particular dictionary, storing encoded messages in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node, retrieving encoded messages for the particular channel from respective buffers having time-to-lives that have not expired and according to the order, decoding each retrieved message based on the particular dictionary, and sending the decoded messages to a plurality of subscriber clients.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving from a plurality of publisher clients a plurality of messages, each message being for a channel of a plurality of channels wherein each channel comprises an ordered plurality of messages;
encoding, by one or more computer processors, each message for a first channel from the plurality of channels based on a dictionary for the first channel, the dictionary defining a pattern associated with each message for the first channel, wherein encoding each message for the first channel comprises compressing the message for the first channel according to the pattern;
storing encoded messages for the first channel in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node;
retrieving encoded messages for the first channel from respective buffers having time-to-lives that have not expired and according to the order;
inspecting content of at least one retrieved encoded message;
determining, by the one or more computer processors, from the content at least one pattern used to encode each retrieved encoded message;
decoding each retrieved encoded message based on the dictionary and the determined pattern, wherein decoding comprises decompressing each message in the first channel according to the determined pattern; and
sending the decoded messages to a plurality of subscriber clients.

2. The method of claim 1 wherein the pattern is shared by at least some of the plurality of messages for the first channel.

3. The method of claim 1 wherein the pattern comprises a text string.

4. The method of claim 1 wherein the pattern corresponds to a common data field shared by at least some of the plurality of messages for the first channel.

5. The method of claim 1 wherein the pattern comprises a data type.

6. The method of claim 1 further comprising: adding the determined pattern to the dictionary for the first channel.

7. The method of claim 1 wherein storing encoded messages for the first channel in one or more respective buffers comprises: sending a plurality of encoded messages to a first buffer on a first node, wherein the first node stores the plurality of encoded messages in a first block of one or more blocks within the first buffer, wherein each block comprises a respective time-to-live.

8. The method of claim 7 wherein retrieving encoded messages for the first channel comprises retrieving encoded messages from one or more of the one or more blocks within the first buffer having respective time-to-lives that have not expired.

9. A system comprising:
a memory; and
one or more computer processors, operatively coupled with the memory, programmed to perform operations to:
receive from a plurality of publisher clients a plurality of messages, each message being for a channel of a plurality of channels wherein each channel comprises an ordered plurality of messages;
encode each message for a first channel from the plurality of channels based on a dictionary for the first channel, the dictionary defining a pattern associated with each message for the first channel, wherein encoding each message for the first channel comprises compressing the message for the first channel according to the pattern;
store encoded messages for the first channel in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node;
retrieve encoded messages for the first channel from respective buffers having time-to-lives that have not expired and according to the order;
inspect content of at least one retrieved encoded message;
determine from the content at least one pattern used to encode each retrieved encoded message;
decode each retrieved encoded message based on the dictionary and the determined pattern, wherein decoding comprises decompressing each message in the first channel according to the determined pattern; and
send the decoded messages to a plurality of subscriber clients.

10. The system of claim 9 wherein the pattern is shared by at least some of the plurality of messages for the first channel.

11. The system of claim 9 wherein the pattern comprises a text string.

12. The system of claim 9 wherein the pattern corresponds to a common data field shared by at least some of the plurality of messages for the first channel.

13. The system of claim 9 wherein the pattern comprises a data type.

14. The system of claim 9 , the operations further to: add the determined pattern to the dictionary for the first channel.

15. The system of claim 9 wherein to store the encoded messages for the first channel in one or more respective buffers the operations are to: send a plurality of encoded messages to a first buffer on a first node, wherein the first node stores the plurality of encoded messages in a first block of one or more blocks within the first buffer wherein each block comprises a respective time-to-live.

16. The system of claim 15 wherein to retrieve the encoded messages for the first channel, the operations are to: retrieve the encoded messages from one or more of the one or more blocks within the first buffer having respective time-to-lives that have not expired.

17. A non-transitory machine-readable medium having instructions stored thereon that, when executed by one or more computer processors, cause the one or more computer processors:
receive from a plurality of publisher clients a plurality of messages, each message being for a channel of a plurality of channels wherein each channel comprises an ordered plurality of messages;
encode each message for a first channel from the plurality of channels based on a dictionary for the first channel, the dictionary defining a pattern associated with each message for the first channel, wherein encoding each message for the first channel comprises compressing the message for the first channel according to the pattern;
store encoded messages for the first channel in one or more respective buffers according to the order, each buffer having a respective time-to-live and residing on a respective node;
retrieve encoded messages for the first channel from respective buffers having time-to-lives that have not expired and according to the order;
inspect content of at least one retrieved encoded message;
determine from the content at least one pattern used to encode each retrieved encoded message;
decode each retrieved encoded message based on the dictionary and the determined pattern, wherein decoding comprises decompressing each message in the first channel according to the determined pattern; and
send the decoded messages to a plurality of subscriber clients.

18. The non-transitory machine-readable medium of claim 17 wherein the pattern is shared at least some of the plurality of messages for the first channel.

19. The non-transitory machine-readable medium of claim 17 wherein the pattern comprises a text string.

20. The non-transitory machine-readable medium of claim 17 wherein the pattern corresponds to a common data field shared by at least some of the plurality of messages for the first channel.

21. The non-transitory machine-readable medium of claim 17 wherein the pattern comprises a data type.

22. The non-transitory machine-readable medium of claim 17 wherein the one or more computer processors are further to: add the determined pattern to the dictionary for the first channel.

23. The non-tr

Assignees

• Machine Zone Inc
• Satori Worldwide Llc

Inventors

• Walkin Lev

Classifications

• H04L69/04

  Protocols for data compression, e.g. ROHC · CPC title

• H04L47/286

  Time to live · CPC title

• H04L51/14

  Electricity · mapped topic

• H04L67/42

  Electricity · mapped topic

• H04L51/066 Primary

  Format adaptation, e.g. format conversion or compression · CPC title