Managing sharable cell-based analytical notebooks

Patent metadata

Field	Value
Publication number	US-10783148-B2
Application number	US-201916357108-A
Country	US
Kind code	B2
Filing date	Mar 18, 2019
Priority date	Aug 18, 2016
Publication date	Sep 22, 2020
Grant date	Sep 22, 2020

Abstract

In an embodiment, a data processing method comprises creating and storing a plurality of analytical notebooks in digital computer storage, wherein each of the analytical notebooks comprises notebook metadata that specifies a kernel for execution, and one or more computational cells, wherein each of the cells comprises cell metadata, a source code reference and an output reference; receiving, in association with a first cell among the one or more cells, first input specifying computer program source code of a function, wherein the function defines an input dataset, a transformation, and one or more variables associated with output data; storing the first cell, excluding the output data, using a first digital data storage system and updating the source code reference to identify the first data storage system; using the kernel specified in the notebook metadata, executing an executable version of the source code to result in generating the output data; storing the output data using a second digital data storage system that is separate from the first digital data storage system and updating the output reference to identify the second data storage system.

First claim

What is claimed is:

1. A method for separately storing data and source code in computer memory, the method comprising:
configuring the computer memory using a data structure architecture for a shareable cell-based computation notebook;
wherein the data structure architecture comprises:
notebook metadata that is to specify a kernel for execution; and
a computational cell;

wherein the computational cell comprises:
cell metadata that is to identify a particular version of source code of a function that defines an input dataset, a transformation, and one or more variables that are to be associated with output data that is to be generated as a result of executing the particular version of the source code;
a source code reference that is to be updated to identify a first storage location that is to contain the particular version of the source code of the function; and
an output reference that is to be updated to identify a second storage location that is to contain the output data that is to be generated as a result of executing the particular version of the source code identified in the cell metadata using the kernel specified in the notebook metadata;
wherein the cell metadata comprises a versionset value representative of a version of a code library or a version of a dataset;
wherein the method is performed by one or more processors.

2. The method of claim 1 , wherein an analytical notebook configured according to the data structure architecture is hosted in a user container of a containerized program execution system in a virtual computing environment.

3. The method of claim 2 , wherein the user container comprises a plurality of different execution kernels.

4. The method of claim 1 , wherein the cell metadata further comprises a library versionset value that represents all program code libraries and all version numbers of the program code libraries on which the source code depends; and a dataset versionset value that represents version values for one or more datasets that the source code specifies as input sources.

5. The method of claim 4 , wherein when the computational cell is connected to one or more datasets having dataset version numbers that do not correspond to the dataset versionset value in the cell metadata, one or more of generating a notification message or blocking execution of the computational cell is performed.

6. The method of claim 1 , wherein the first storage location comprises a first digital data storage system.

7. The method of claim 6 , wherein the second storage location is separate from the first storage location.

8. The method of claim 1 wherein the notebook metadata specifies any of R, PYTHON or MATLAB as the kernel for execution.

9. A method for separately storing data and source code in computer memory, the method comprising:
configuring the computer memory using a data structure architecture for a shareable cell-based computation notebook;
wherein the data structure architecture comprises:
notebook metadata that is to specify a kernel for execution; and
a computational cell;

wherein the computational cell comprises:
cell metadata that is to identify a particular version of source code of a function that defines an input dataset, a transformation, and one or more variables that are to be associated with output data that is to be generated as a result of executing the particular version of the source code;
a source code reference that is to be updated to identify a first storage location that is to contain the particular version of the source code of the function; and
an output reference that is to be updated to identify a second storage location that is to contain the output data that is to be generated as a result of executing the particular version of the source code identified in the cell metadata using the kernel specified in the notebook metadata;
wherein the cell metadata further comprises a library versionset value that represents all program code libraries and all version numbers of the program code libraries on which the source code depends;
wherein when the computational cell is connected to program code libraries having version numbers that do not correspond to the library versionset value in the cell metadata, one or more notification messages blocking execution of the computational cell are generated.

10. A method for separately storing data and source code in computer memory, the method comprising:
configuring the computer memory using a data structure architecture for a shareable cell-based computation notebook;
wherein the data structure architecture comprises:
notebook metadata that is to specify a kernel for execution; and
a computational cell;

wherein the computational cell comprises:
cell metadata that is to identify a particular version of source code of a function that defines an input dataset, a transformation, and one or more variables that are to be associated with output data that is to be generated as a result of executing the particular version of the source code;
a source code reference that is to be updated to identify a first storage location that is to contain the particular version of the source code of the function; and
an output reference that is to be updated to identify a second storage location that is to contain the output data that is to be generated as a result of executing the particular version of the source code identified in the cell metadata using the kernel specified in the notebook metadata; and the method further comprises:
in response to execution of the source code of the computational cell, determining names of variables that are then currently in memory representing a local scope of the computational cell and obtaining then-current values of the variables;
generating and displaying a view of the names of the variables and the then-current values of the variables in a user interface that also shows the computational cell.

11. A computer system comprising:
one or more processors;
one or more non-transitory computer-readable storage media storing instructions which, when executed by the one or more processors, cause the one or more processors to perform:

configuring a computer memory using a data structure architecture for a shareable cell-based computation notebook;
wherein the data structure architecture comprises:
notebook metadata that is to specify a kernel for execution; and
a computational cell;

wherein the computational cell comprises:
cell metadata that is to identify a particular version of source code of a function that defines an input dataset, a transformation, and one or more variables that are to be associated with output data that is to be generated as a result of executing the particular version of the source code;
a source code reference that is to be updated to identify a first storage location that is to contain the particular version of the source code of the function; and
an output reference that is to be updated to identify a second storage location that is to contain the output data that is to be generated as a result of executing the particular version of the source code identified in the cell metadata using the kernel specified in the notebook metadata;
wherein the cell metadata comprises a versionset value representative of a version of a code library or a version of a dataset.

12. The computer system of claim 11 , wherein an analytical notebook configured according to the data structure architecture is hosted in a user container of a containerized program execution system in a virtual computing environment.

13. The computer system of claim 12 , wherein the user container co

Assignees

• Palantir Technologies Inc

Inventors

• Shuster James
• Fernandez Daniel

Classifications

• G06F8/20 Primary

  Software design · CPC title

• G06F2009/45587

  Isolation or security of virtual machine instances · CPC title

• G06F2009/45579

  I/O management, e.g. providing access to device drivers or storage · CPC title

• G06F40/18

  of spreadsheets (form-filling G06F40/174) · CPC title

• G06F21/604

  Tools and structures for managing or administering access control systems · CPC title