Callable notebook for cluster execution

Patent metadata

Field	Value
Publication number	US-10678536-B2
Application number	US-201916378353-A
Country	US
Kind code	B2
Filing date	Apr 8, 2019
Priority date	Jan 29, 2016
Publication date	Jun 9, 2020
Grant date	Jun 9, 2020

Abstract

A system for processing a notebook includes an input interface and a processor. The input interface is to receive a first notebook. The notebook comprises code for interactively querying and viewing data. The processor is to load the first notebook into a shell. The shell receives one or more parameters associated with the first notebook. The shell executes the first notebook using a cluster.

First claim

What is claimed is:

1. A system for processing a notebook, comprising:
an input interface to receive a first notebook, wherein the notebook comprises code for interactively querying and viewing data; and
a processor to:
load the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more parameters associated with the first notebook; and
execute a command of the at least one command, comprising to:
determine whether a selected command relates to a local command; and
in response to a determination that the selected command does not relate to the local command:
determine whether the selected command relates to a cluster command, comprising to:
 determine whether the selected command relates to a cluster command based on an estimated time to complete the command locally, an estimated processor load to complete the command locally, a data usage, an indication of whether the command is the cluster command, or any combination thereof; and
in response to a determination that the selected command does not relate to the cluster command:
 determine that the selected command relates to a special case; and
 execute the special case command.

2. The system of claim 1 , wherein the shell replaces one or more variables in the first notebook with the one or more input parameters.

3. The system of claim 1 , wherein the shell executes the first notebook a second time using one or more different input parameters.

4. The system of claim 1 , wherein the first notebook comprises a command to execute a second notebook.

5. The system of claim 4 , wherein the second notebook and the first notebook comprise different notebooks.

6. The system of claim 4 , wherein the second notebook is written in a different language than the first notebook.

7. The system of claim 4 , wherein the second notebook and the first notebook comprise the same notebook.

8. The system of claim 4 , wherein the second notebook produces output.

9. The system of claim 1 , wherein the first notebook produces output.

10. The system of claim 1 , wherein the shell is configured to prevent recursion.

11. The system of claim 1 , wherein the processor provides an indication for display based at least in part on an output from the first notebook.

12. The system of claim 1 , wherein the first notebook comprises multiple code sections.

13. The system of claim 12 , wherein code sections within the first notebook are in different languages.

14. The system of claim 1 , wherein the shell re-executes the first notebook.

15. The system of claim 1 , wherein the shell re-executes a code section of the first notebook.

16. The system of claim 1 , wherein executing the first notebook comprises accessing a big data repository.

17. The system of claim 16 , wherein updated data is accessed from the big data repository.

18. The system of claim 17 , wherein the shell re-executes the first notebook using the updated data.

19. A method for processing a notebook, comprising:
receiving a first notebook, wherein the notebook comprises code for interactively querying and viewing data;
loading, using a processor, the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more parameters associated with the first notebook; and
executing a command of the at least one command, comprising:
determining whether a selected command relates to a local command; and
in response to a determination that the selected command does not relate to the local command:
determining whether the selected command relates to a cluster command, comprising:
determining whether the selected command relates to a cluster command based on an estimated time to complete the command locally, an estimated processor load to complete the command locally, a data usage, an indication of whether the command is the cluster command, or any combination thereof; and

in response to a determination that the selected command does not relate to the cluster command:
determining that the selected command relates to a special case; and
executing the special case command.

20. A computer program product for processing a notebook, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:
receiving a first notebook, wherein the notebook comprises code for interactively querying and viewing data;
loading the first notebook into a shell, wherein the first notebook includes at least one command, and wherein the shell receives one or more parameters associated with the first notebook; and
executing a command of the at least one command, comprising:
determining whether a selected command relates to a local command; and
in response to a determination that the selected command does not relate to the local command:
determining whether the selected command relates to a cluster command, comprising:
determining whether the selected command relates to a cluster command based on an estimated time to complete the command locally, an estimated processor load to complete the command locally, a data usage, an indication of whether the command is the cluster command, or any combination thereof; and

in response to a determination that the selected command does not relate to the cluster command:
determining that the selected command relates to a special case; and
executing the special case command.

Assignees

• Databricks Inc

Inventors

• Hunter Timothee
• Ghodsi Ali
• Stoica Ion

Classifications

• G06F16/9535

  Search customisation based on user profiles and personalisation · CPC title

• G06F9/5066

  Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451) · CPC title

• G06F8/71 Primary

  Version control (security arrangements therefor G06F21/57); Configuration management · CPC title

• G06F8/54

  Link editing before load time · CPC title

• G06F9/45512

  Command shells · CPC title