Managing sharable cell-based analytical notebooks

Patent metadata

Field	Value
Publication number	US-11226967-B2
Application number	US-202016994276-A
Country	US
Kind code	B2
Filing date	Aug 14, 2020
Priority date	Aug 18, 2016
Publication date	Jan 18, 2022
Grant date	Jan 18, 2022

Abstract

In an embodiment, a data processing method comprises creating and storing a plurality of analytical notebooks in digital computer storage, wherein each of the analytical notebooks comprises notebook metadata that specifies a kernel for execution, and one or more computational cells, wherein each of the cells comprises cell metadata, a source code reference and an output reference; receiving, in association with a first cell among the one or more cells, first input specifying computer program source code of a function, wherein the function defines an input dataset, a transformation, and one or more variables associated with output data; storing the first cell, excluding the output data, using a first digital data storage system and updating the source code reference to identify the first data storage system; using the kernel specified in the notebook metadata, executing an executable version of the source code to result in generating the output data; storing the output data using a second digital data storage system that is separate from the first digital data storage system and updating the output reference to identify the second data storage system.

First claim

What is claimed is:

1. A method, comprising:
configuring a computer memory using a data structure architecture for a shareable cell-based computation notebook, the data structure architecture comprising notebook metadata specifying a kernel for execution;
in response to receiving a selection and movement of a graphical user interface element, creating a computational cell and adding the computational cell to the notebook,
the computational cell comprising cell metadata, a source code reference, and an output reference;
the cell metadata identifying a particular version of source code of a function that defines an input dataset, a transformation, and one or more variables that are to be associated with output data that is to be generated as a result of executing the particular version of the source code;
updating the source code reference to identify a first storage location that is to contain the particular version of the source code of the function;
updating the output reference to identify a second storage location that is to contain the output data that is to be generated as a result of executing the particular version of the source code identified in the cell metadata using the kernel specified in the notebook metadata,
wherein the method is performed by one or more computing devices.

2. The method of claim 1 , the selection comprising selecting an icon, the selection of the icon causing the computational cell to be created.

3. The method of claim 1 , the selection comprising selecting a widget control and moving a representation of a widget, the selection of the widget control and movement of the representation of the widget causing the computational cell to be added to the notebook.

4. The method of claim 1 , wherein the first storage location and the second storage location are subject to different access control systems.

5. The method of claim 4 , further comprising, in response to adding the function to the computational cell, creating function source code and output data, and storing the function source code and output data in separate locations.

6. The method of claim 1 , further comprising, in response to input, generating and displaying a view of a variable that has been defined for the function of the computational cell.

7. The method of claim 1 , further comprising creating and storing at least one versionset value associated with the function.

8. The method of claim 1 , further comprising, in response to input, rendering a graphical representation of the computational cell, the graphical representation comprising a plurality of widgets representing variables defined the source code and a plurality of input boxes configured to accept data corresponding to the variables.

9. The method of claim 8 , further comprising, in response to acceptance of data by a widget of the plurality of widgets, providing the data to the computational cell, and producing, by the computational cell executing the source code, output based on the provided data.

10. The method of claim 1 , an analytical notebook configured according to the data structure architecture capable of being hosted in a user container of a containerized program execution system in a virtual computing environment, the user container comprising a plurality of different execution kernels.

11. A computer system comprising:
one or more processors;
one or more non-transitory computer-readable storage media storing instructions which, when executed by the one or more processors, cause the one or more processors to be capable of performing:
configuring a computer memory using a data structure architecture for a shareable cell-based computation notebook, the data structure architecture comprising notebook metadata specifying a kernel for execution;
by a selection and movement of a graphical user interface element, creating a computational cell and adding the computational cell to the notebook;
the computational cell comprising cell metadata, a source code reference, and an output reference;
the cell metadata identifying a particular version of source code of a function that defines an input dataset, a transformation, and one or more variables that are to be associated with output data that is to be generated as a result of executing the particular version of the source code;
the source code reference being updated to identify a first storage location that is to contain the particular version of the source code of the function;
the output reference being updated to identify a second storage location that is to contain the output data that is to be generated as a result of executing the particular version of the source code identified in the cell metadata using the kernel specified in the notebook metadata;
wherein the method is performed by one or more computing devices.

12. The system of claim 11 , the selection comprising selecting an icon, the selection of the icon causing the computational cell to be created.

13. The system of claim 11 , the selection comprising selecting a widget control and moving a representation of a widget, the selection of the widget control and movement of the representation of the widget causing the computational cell to be added to the notebook.

14. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause the one or more processors to be capable of, using an editor, adding the function to the computational cell.

15. The system of claim 14 , further comprising instructions which, when executed by the one or more processors, cause the one or more processors to be capable of, in response to adding the function to the computational cell, creating function source code and output data, and storing the function code and output data in separate locations.

16. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause the one or more processors to be capable of, in response to input, generating and displaying a view of a variable that has been defined for the function of the computational cell.

17. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause the one or more processors to be capable of creating and storing at least one versionset value associated with the function.

18. The system of claim 11 , further comprising instructions which, when executed by the one or more processors, cause the one or more processors to be capable of, in response to input, rendering a graphical representation of the computational cell, the graphical representation comprising a plurality of widgets representing variables defined the source code and a plurality of input boxes configured to accept data corresponding to the variables.

19. The method of claim 18 , further comprising, in response to acceptance of data by a widget of the plurality of widgets, providing the data to the computational cell, and producing, by the computational cell executing the source code, output based on the provided data.

20. The method of claim 11 , an analytical notebook configured according to the data structure architecture capable of being hosted in a user container of a containerized program execution system in a virtual computing environment, the user container comprising a plurality of different execution kernels.

Assignees

• Palantir Technologies Inc

Inventors

• Shuster James
• Fernandez Daniel

Classifications

• G06F8/20 Primary

  Software design · CPC title

• G06F8/38

  for implementing user interfaces · CPC title

• G06F9/30043

  LOAD or STORE instructions; Clear instruction · CPC title

• G06F8/33

  Intelligent editors · CPC title

• G06F8/313

  Logic programming, e.g. PROLOG programming language · CPC title