Publish-subscribe platform for cloud file distribution

Patent metadata

Field	Value
Publication number	US-2017109369-A1
Application number	US-201615335302-A
Country	US
Kind code	A1
Filing date	Oct 26, 2016
Priority date	Mar 13, 2012
Publication date	Apr 20, 2017
Grant date	—

Abstract

Methods and apparatus for managing file distribution between publishing and subscribing devices are provided. At a transporter system, a publish request for publishing a version of the dataset is received from a publisher device, wherein the publish request has a predefined format that includes an identity of the dataset, an identity of the version of the dataset, and an identity of a location for the version of the dataset. The transporter system updates a registry to associate the dataset with the identity of the dataset, version, and location, and the transporter system automatically publishes the dataset to any subscribers devices that have previously sent subscribe requests for such dataset to the transporter system.

First claim

What is claimed is:

1 . A computer implemented method of managing file distribution between publishing and subscribing devices, comprising:
at a transporter system, receiving from a publisher device a publish request for publishing a version of a dataset, wherein the publish request has a predefined format that includes an identity of the dataset, an identity of the version of the dataset, and an identity of a location for the version of the dataset; the transporter system updating a registry to associate the dataset with the identity of the dataset, version, and location; and the transporter system automatically publishing the version of the dataset to any subscriber devices that have previously sent subscribe requests for such dataset to the transporter system.

2 . The method of claim 1 , wherein the publisher device has no knowledge of any subscriber devices to the dataset, and any subscriber devices to such dataset have no knowledge of the publisher device, the method further comprising:
the transporter system generating a checksum for the dataset; the transporter system updating the registry to associate the generated checksum with the dataset; and using the checksum to validate the version of the dataset prior to automatically publish the version of the dataset, wherein the version of the dataset is only automatically published if the versioned dataset is valid.

3 . The method of claim 1 , further comprising the transporter system storing the dataset in a cloud storage network, and wherein the transporter system is scalable along multiple dimensions, including dataset size, number of datasets, and number of subscribers to a dataset, and wherein automatically publishing includes the transporter system publishing to any subscribers directly or through multiple tiered distributors.

4 . The method of claim 1 , wherein the predefined format of the publish request further includes information specifying how long the versioned dataset is to retained by the transporter system and how many versions of the dataset are to be retained by the transporter system, and metadata specifying custom key/value pairs to be associated with the versioned dataset, the method further comprising:
determining whether the publish request is for a duplicate version of the dataset that has already been published; and sending an error message to the publisher device if the publication request is for a duplicate version, and wherein the operation for updating and automatically publishing are only performed if the version of the publish request is determined to not be a duplicate version.

5 . The method of claim 1 , further comprising:
at a transporter system, receiving from one or more subscribing devices a subscribe request for the dataset, wherein the subscribe request has a predefined format that includes the identity of the dataset and a destination identity for receiving published updates of the dataset; and after receiving each subscribe request, the transporter system updating the registry to associate the destination identity of such subscribe request with the identity of the dataset.

6 . The method of claim 5 , further comprising:
repeating the operations for receiving from a publisher device, updating the registry, and automatically publishing for a plurality of publish requests for different datasets and different dataset versions from a plurality of different publisher devices, wherein at least one dataset is published by notifying a plurality of associated subscribers of individual chunks of such dataset so that such subscribers can share the individual chunks with each other; repeating the operations for receiving from one or more subscriber devices and updating the registry for a plurality of subscribe requests for the different datasets; if a query request to discover dataset versions is received, providing information from the registry for one or more datasets to a sender of such query call; and if a remove request for a particular identified dataset version is received from a publisher device, removing information for the particular identified dataset version from the registry so that the particular identified dataset version is not published to any subscribers.

7 . The method of claim 6 , wherein the predefined format of the publish request further includes a namespace identity that is also associated with the dataset in the registry and the predefined format of each subscribe request for the dataset also includes the namespace identity, an executable and associated arguments for a command to be executed for each updated version of the dataset that is published, wherein the destination identity specifies a host device and a consumer identity, the method further comprising:
for each automatically published updated version of each dataset, automatically invoking the command associated with such dataset, wherein the command is invoked at the specified host device.

8 . An apparatus comprising at least a processor and a memory, wherein the processor and/or memory are configured to perform the following operations:
receiving from a publisher device a publish request for publishing a version of a dataset, wherein the publish request has a predefined format that includes an identity of the dataset, an identity of the version of the dataset, and an identity of a location for the version of the dataset; updating a registry to associate the dataset with the identity of the dataset, version, and location; and automatically publishing the version of the dataset to any subscribers devices that have previously subscribed to such dataset.

9 . The apparatus of claim 8 , wherein the publisher device has no knowledge of any subscriber devices to the dataset, and any subscriber devices to such dataset have no knowledge of the publisher device, wherein the processor and/or memory are further configured for:
generating a checksum for the dataset; updating the registry to associate the generated checksum with the dataset; and using the checksum to validate the version of the dataset prior to automatically publish the version of the dataset, wherein the version of the dataset is only automatically published if the version of the dataset is valid.

10 . The apparatus of claim 8 , wherein the processor and/or memory are further configured for storing the dataset in a cloud storage network, and wherein the transporter system is scalable along multiple dimensions, including dataset size, number of datasets, and number of subscribers to a dataset, and wherein automatically publishing includes the transporter system publishing to any subscribers directly or through multiple tiered distributors.

11 . The apparatus of claim 8 , wherein the predefined format of the publish request further includes information specifying how long the version of the dataset is to retained and how many versions of the dataset are to be retained, and metadata specifying custom key/value pairs to be associated with the version of the dataset, wherein the processor and/or memory are further configured for:
determining whether the publish request is for a duplicate version of the dataset that has already been published; and sending an error message to the publisher device if the publication request is for a duplicate version, and wherein the operation for updating and automatically publishing are only performed if the version of the publish request is determined to not be a duplicate version.

12 . The apparatus of claim 8 , wherein the processor and/or memory are further configured for:
receiving from one or more subscribing devices a subscribe request for the dataset, wherein the subscribe

Assignees

• Yahoo Inc

Inventors

• Feng Andrew An
• Chandra Rohit
• Suryanarayanan Lakshmanan
• Crowder Timothy R
• Lam Victor J

Classifications

• H04L67/1097

  for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] · CPC title

• G06F17/30144

  Physics · mapped topic

• H04L67/06 Primary

  specially adapted for file transfer, e.g. file transfer protocol [FTP] · CPC title

• G06F17/30174 Primary

  Physics · mapped topic

• G06F11/1004

  to protect a block of data words, e.g. CRC or checksum (G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00) · CPC title