Secure multi-directional data pipeline for data distribution systems

Patent metadata

Field	Value
Publication number	US-11012500-B2
Application number	US-201816047968-A
Country	US
Kind code	B2
Filing date	Jul 27, 2018
Priority date	Jul 27, 2018
Publication date	May 18, 2021
Grant date	May 18, 2021

Abstract

Techniques for ingesting data streams to a distributed-computing system using a multi-directional data ingestion pipeline are provided. In one embodiment, a method for ingesting data streams includes, at a client gateway, receiving a plurality of messages; assigning the plurality of messages to one or more data streams; obtaining stream routing configurations; and identifying one or more receivers. The method further includes determining whether at least one of the one or more data streams is to be delivered to one or more receivers operating in the first computing environment; and if so, delivering the at least one of the one or more data streams to the one or more receivers operating in the first computing environment. The method further includes delivering the one or more data streams to a data ingress gateway operating in a second computing environment.

First claim

What is claimed is:

1. A method for stream processing of one or more data streams ingested from a client gateway using a multi-directional data ingestion pipeline, the method comprising:
at a data ingress gateway operating in a second computing environment having one or more processors and memory:
receiving a first data stream ingested from a client gateway operating in a first computing environment different from the second computing environment;
obtaining, based on the first data stream and receiver registration information stored within the data ingress gateway, a first delivery policy associated with a first receiver group including one or more receivers;
receiving a second data stream ingested from the client gateway, wherein the second data stream is different from the first data stream;
obtaining, based on the second data stream and the receiver registration information, a second delivery policy associated with a second receiver group including one or more receivers, wherein the second delivery policy is different from the first delivery policy;
delivering the first data stream to the first receiver group in accordance with the first delivery policy; and
delivering the second data stream to the second receiver group in accordance with the second delivery policy.

2. The method of claim 1 , wherein the first computing environment is a private network associated with a client of the distributed-computing system and the second computing environment is a network associated with a service provider that provides data services to the client, the private network associated with the client and the network associated with the service provider being separate networks.

3. The method of claim 1 , wherein the receiver registration information is included in a routing table within the data ingress gateway.

4. The method of claim 1 , wherein obtaining, based on the first data stream and the receiver registration information, the first delivery policy associated with the first receiver group comprises:
identifying destination information included in one or more messages of the first data stream;
identifying one or more receivers included in the first receiver group based on the destination information, wherein the one or more receivers included in the first receiver group are registered with the data ingress gateway; and
obtaining the first delivery policy, wherein the first delivery policy is associated with the identified one or more receivers included in the first receiver group.

5. The method of claim 1 , wherein obtaining, based on the second data stream and the receiver registration information, the second delivery policy associated with the second receiver group including one or more receivers comprises:
identifying destination information included in one or more messages of the second data stream;
identifying one or more receivers included in the second receiver group based on the destination information, wherein the one or more receivers included in the second receiver group are registered with the data ingress gateway; and
obtaining the second delivery policy, wherein the second delivery policy is associated with the identified one or more receivers included in the second receiver group.

6. The method of claim 1 , wherein the first delivery policy or the second delivery policy comprises one of:
waiting for all receivers in the first receiver group or the second receiver group, respectively, to respond to the data ingress gateway with OK statuses before the data ingress gateway responds to the client gateway;
waiting for one of the receivers in the first receiver group or the second receiver group, respectively, to respond to the data ingress gateway with an OK status before the data ingress gateway responds to the client gateway; and
the data ingress gateway responds to the client gateway without waiting for a receiver in the first receiver group or the second receiver group, respectively, to respond with OK statuses to the data ingress gateway.

7. The method of claim 1 , wherein delivering the first data stream to the first receiver group in accordance with the first delivery policy comprises:
identifying destination information included in one or more messages of the first data stream;
obtaining routing information stored in stream routing configurations associated with the data ingress gateway;
routing, based on the routing information and the destination information, the first data stream to one or more receivers of the first receiver group; and
responding to the client gateway in accordance with the first delivery policy.

8. The method of claim 7 , wherein routing the first data stream to one or more receivers of the first receiver group comprises delivering the first data stream to a service agent of the first receiver group.

9. The method of claim 7 , wherein the one or more receivers of the first receiver group are publish-subscribe receivers, and wherein,
the first data stream is stored in at least one of the publish-subscribe receivers under a corresponding topic, and
each of one or more subscribers of the first data stream obtains data from the at least one of the publish-subscribe receivers based on one or more subscribed topics.

10. The method of claim 1 , wherein delivering the second data stream to the second receiver group in accordance with the second delivery policy comprises:
identifying destination information included in one or more messages of the second data stream;
obtaining routing information stored in stream routing configurations associated with the data ingress gateway;
routing, based on the routing information and the destination information, the second data stream to one or more receivers of the second receiver group; and
responding to the client gateway in accordance with the second delivery policy.

11. The method of claim 10 , wherein routing the second data stream to one or more receivers of the second receiver group comprises delivering the second data stream to a service agent of the second receiver group.

12. The method of claim 10 , wherein the one or more receivers of the second receiver group are publish-subscribe receivers, and wherein,
the second data streams is stored in at least one of the publish-subscribe receivers under a corresponding topic, and
each of one or more subscribers of the second data stream obtains data from the at least one of the publish-subscribe receivers based on one or more subscribed topics.

13. The method of claim 1 , wherein delivering the first data stream and delivering the second data stream are performed in accordance with an order corresponding to an order of receiving the first data stream and the second data stream.

14. The method of claim 1 , wherein delivering the first data stream and delivering the second data stream are performed in accordance with a predefine order determined based on priorities associated with one or more subscribers of the first data stream and priorities the associated with one or more subscribers of the second data stream.

15. The method of claim 1 , wherein delivering the first data stream and delivering the second data stream are performed dynamically based on one or more network-related conditions.

16. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by a data ingress gateway operating in a second computing environment having one or more processors and memory, the one or more programs including instructions for:
processing a first data stream ingested from a client gateway operating in a first computing e

Assignees

• Vmware Inc

Inventors

• Dar Sufian A
• Curtis Tyler J
• Cooley James W
• Beg Mirza Mohsin

Classifications

• H04L67/63

  Routing a service request depending on the request content or context · CPC title

• H04L45/38

  Flow based routing · CPC title

• G06F9/45558 Primary

  Hypervisor-specific management and integration aspects · CPC title

• H04L67/10 Primary

  in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) · CPC title

• H04L63/08

  for authentication of entities (cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32) · CPC title