Decentralized data platform

Patent metadata

Field	Value
Publication number	US-11886913-B2
Application number	US-202117354200-A
Country	US
Kind code	B2
Filing date	Jun 22, 2021
Priority date	Jun 22, 2021
Publication date	Jan 30, 2024
Grant date	Jan 30, 2024

Abstract

Data from data sources may be processed at an edge device. The edge device may generate a local processing result, filter the data, and/or prioritize the data. Accordingly, data is transmitted from the edge device to the data platform, where it may be processed further. For example, a local processing result may be processed at the data platform, such that processing is performed without all of the data source data. In examples, at least a part of such data may remain at an edge device. The edge device may maintain a manifest of data stored by the edge device. The data platform may generate an aggregated manifest using manifests from associated edge devices, such that it may be determined where data is stored. As a result, the data platform may redirect requests to an associated edge device when it is determined that requested data is remote from the data platform.

First claim

What is claimed is:

1. A system comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, causes the system to perform a set of operations, the set of operations comprising:
receiving, by the system, data from a data source;
processing the data to generate a local processing result;
generating a manifest indicating an association between a part of the received data and the system;
providing, to a data platform, the local processing result and the manifest;
providing the part of the received data to the data platform; and
generating an updated manifest that omits the association between the part of the received data and the system.

2. The system of claim 1 , wherein:
processing the data comprises identifying a first group of data comprising the local processing result and a second group of data comprising the part of the received data; and
the first group of data has a higher priority than the second group of data.

3. The system of claim 2 , wherein:
the first group of data is provided to the data platform substantially contemporaneously with receipt of the data from the data source; and
the second group of data is provided to the data platform in response to identifying computing resource availability.

4. The system of claim 1 , wherein the set of operations further comprises:
receiving, from the data platform, a platform processing result; and
updating the manifest to indicate an association between the platform processing result and the system.

5. The system of claim 4 , wherein the platform processing result is associated with the local processing result and data of another edge device.

6. The system of claim 1 , wherein the set of operations further comprises:
receiving, from a computing device, a request for data comprising authentication information;
validating the request for data using the authentication information; and
when the request for data is validated:
identifying, using the manifest, response data associated with the request for data; and
providing, in response to the request for data, the identified response data.

7. The system of claim 6 , wherein the authentication information comprises an authorization token associated with the data platform.

8. A method for ingesting data from an edge device associated with a data platform, the method comprising:
receiving, from a first edge device, first data and a first manifest;
receiving, from a second edge device, second data and a second manifest;
generating an aggregated manifest based on the first manifest and the second manifest, wherein the aggregated manifest comprises:
an association between a first part of data and the first device; and
an association between a second part of data and the second device;

generating a platform processing result based on the first data and the second data;
providing, to a recipient edge device that is the first edge device or the second edge device, the platform processing result; and
generating an updated aggregated manifest to comprise an association between the platform processing result and the recipient edge device.

9. The method of claim 8 , further comprising:
receiving, from a computing device, a request for data of the data platform;
determining, based on the updated aggregated manifest, a storage location of requested data associated with the request; and
when it is determined the storage location is the data platform, providing at least a part of the requested data in response to the request.

10. The method of claim 8 , further comprising:
receiving, from a computing device, a request for data of the data platform;
determining, based on the updated aggregated manifest, a storage location of requested data associated with the request; and
when it is determined the storage location is an edge device:
generating authentication information for the computing device; and
providing, in response to the request, a redirection indication comprising the authentication information and an indication of the edge device.

11. The method of claim 10 , wherein generating the authentication information comprises communicating with the edge device to generate an authorization token.

12. The method of claim 8 , further comprising:
receiving, from the first edge device, third data having a lower priority than the first data; and
updating the manifest to omit an association between the third data and the first edge device.

13. The method of claim 8 , wherein the first data and the second data each have the same priority.

14. A method for processing data by an edge device of a data platform, the method comprising:
receiving, by the edge device, data from a data source;
processing the data to generate a local processing result;
generating a manifest indicating an association between a part of the received data and the edge device;
providing, to a data platform, the local processing result and the manifest;
providing the part of the received data to the data platform; and
generating an updated manifest that omits the association between the part of the received data and the edge device.

15. The method of claim 14 , wherein:
processing the data comprises identifying a first group of data comprising the local processing result and a second group of data comprising the part of the received data; and
the first group of data has a higher priority than the second group of data.

16. The method of claim 15 , wherein:
the first group of data is provided to the data platform substantially contemporaneously with receipt of the data from the data source; and
the second group of data is provided to the data platform in response to identifying computing resource availability.

17. The method of claim 14 , further comprising:
receiving, from the data platform, a platform processing result; and
updating the manifest to indicate an association between the platform processing result and the edge device.

18. The method of claim 17 , wherein the platform processing result is associated with the local processing result and data of another edge device.

19. The method of claim 14 , further comprising:
receiving, from a computing device, a request for data comprising authentication information;
validating the request for data using the authentication information; and
when the request for data is validated:
identifying, using the manifest, response data associated with the request for data; and
providing, in response to the request for data, the identified response data.

20. The method of claim 19 , wherein the authentication information comprises an authorization token associated with the data platform.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Umay Mehmet Kadri
• Badam Anirudh
• Witte Philipp Andre
• Siddique Imran

Classifications

• G06F9/4881 Primary

  Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues · CPC title

• G06F16/285

  Clustering or classification · CPC title

• G06F21/6218

  to a system of files or objects, e.g. local or distributed file system or database · CPC title

• G06F16/256 Primary

  in federated or virtual databases · CPC title