Callable notebook for cluster execution

Patent metadata

Field	Value
Publication number	US-2018121194-A1
Application number	US-201715803604-A
Country	US
Kind code	A1
Filing date	Nov 3, 2017
Priority date	Jan 29, 2016
Publication date	May 3, 2018
Grant date	—

Abstract

A system for processing a notebook includes an input interface and a processor. The input interface is to receive a first notebook. The notebook comprises code for interactively querying and viewing data. The processor is to load the first notebook into a shell. The shell receives one or more parameters associated with the first notebook. The shell executes the first notebook using a cluster.

First claim

What is claimed is:

1 . (canceled)

2 . A system for processing a notebook, comprising:
an input interface to receive a first notebook, wherein the notebook comprises code for interactively querying and viewing data; and a processor to:
load the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more input parameters associated with the first notebook; and
execute a command of the at least one command, comprising to:
determine whether the command relates to an execute notebook command; and
in response to a determination that the command relates to the execute notebook command, execute at least one notebook.

3 . The system of claim 2 , wherein the shell replaces one or more variables in the first notebook with the one or more input parameters.

4 . The system of claim 2 , wherein the shell executes the first notebook a second time using one or more different input parameters.

5 . The system of claim 2 , wherein the first notebook comprises a command to execute a second notebook.

6 . The system of claim 5 , wherein the second notebook and the first notebook comprise different notebooks.

7 . The system of claim 5 , wherein the second notebook is written in a different language than the first notebook.

8 . The system of claim 5 , wherein the second notebook and the first notebook comprise the same notebook.

9 . The system of claim 5 , wherein the second notebook produces output.

10 . The system of claim 2 , wherein the first notebook produces output.

11 . The system of claim 2 , wherein the shell is configured to prevent recursion.

12 . The system of claim 2 , wherein the processor provides an indication for display based at least in part on an output from the first notebook.

13 . The system of claim 2 , wherein the first notebook comprises multiple code sections.

14 . The system of claim 13 , wherein code sections within the first notebook are in different languages.

15 . The system of claim 2 , wherein the shell re-executes the first notebook.

16 . The system of claim 2 , wherein the shell re-executes a code section of the first notebook.

17 . The system of claim 2 , wherein executing the first notebook comprises accessing a big data repository.

18 . The system of claim 17 , wherein updated data is accessed from the big data repository.

19 . The system of claim 18 , wherein the shell re-executes the first notebook using the updated data.

20 . A method for processing a notebook, comprising:
receiving a first notebook, wherein the notebook comprises code for interactively querying and viewing data; and loading the first notebook into a shell using a processor, wherein the first notebook includes at least one command, and wherein the shell receives one or more input parameters associated with the first notebook; and executing a command of the at least one command, comprising:
determining whether the command relates to an execute notebook command; and
in response to a determination that the command relates to the execute notebook command, executing at least one notebook.

21 . A computer program product for processing a notebook, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a first notebook, wherein the notebook comprises code for interactively querying and viewing data; and loading the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more input parameters associated with the first notebook; and executing a command of the at least one command, comprising:
determining whether the command relates to an execute notebook command; and
in response to a determination that the command relates to the execute notebook command, executing at least one notebook.

Assignees

• Databricks Inc

Inventors

• Hunter Timothee
• Ghodsi Ali
• Stoica Ion

Classifications

• G06F9/45512

  Command shells · CPC title

• G06F8/71 Primary

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F9/445 Primary

  Program loading or initiating (bootstrapping G06F9/4401; security arrangements for program loading or initiating G06F21/57) · CPC title

• G06F9/45504

  Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators · CPC title

• G06F8/54

  Link editing before load time · CPC title