Jupyter
From FarmShare
m (→FarmShare Specific Features: link explaining encryption and link to python) |
(default virtual environment) |
||
Line 13: | Line 13: | ||
* [https://www.python.org python3] support | * [https://www.python.org python3] support | ||
- | + | == Installation == | |
+ | Jupyter setup is best performed via the [https://en.wikipedia.org/wiki/Linux Linux] console. This tutorial will be mostly step-by-step, but it general familiarity with the Linux console is helpful. Useful Linux console tutorials are available as a [http://linuxcommand.org/tlcl.php free book], [http://linuxcommand.org/lc3_learning_the_shell.php online tutorial], or [https://www.youtube.com/playlist?list=PLII6oL6B7q78PKy6_R6JTkkYjVXZBZcVq video series]. | ||
+ | === SSH into FarmShare === | ||
+ | [https://en.wikipedia.org/wiki/Secure_Shell SSH] is a protocol that enables an encrypted remote [https://en.wikipedia.org/wiki/Shell_(computing) shell] into another system. SSH is the primary way that Jupyter will be installed and accessed. SSH into one of the farmshare computers. The corn cluster (corn.stanford.edu) is a good choice. Windows does not come with an SSH client, but [http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html putty] is a a free and lightweight SSH client for Windows. Mac and Linux have ssh clients accessible via the terminal. | ||
+ | |||
+ | === Bind to Jupyter virtual environment === | ||
+ | |||
+ | <source lang="sh"> | ||
+ | corn01:~> bash | ||
+ | jane@corn01:~$ source /afs/ir.stanford.edu/group/bil/jupyter/bin/activate | ||
+ | (jupyter)jane@corn01:~$ | ||
+ | </source> | ||
+ | |||
+ | These two commands switch the shell to [https://en.wikipedia.org/wiki/Bash_(Unix_shell) bash] and then update the [http://tldp.org/LDP/Bash-Beginners-Guide/html/chap_03.html environment] to use a pre-built Jupyter installation. | ||
+ | |||
+ | This Jupyter environment has the following packages installed: | ||
+ | |||
+ | <code> | ||
+ | bokeh ipython jupyter matplotlib numpy pandas scipy seaborn statsmodels sympy | ||
+ | </code> | ||
+ | |||
+ | === Create a Jupyter virtual environment === | ||
+ | |||
+ | Although farmshare has python installed, jupyter is not yet installed/supported. Since users do not have administrative access, | ||
Revision as of 15:11, 23 December 2016
Contents |
Jupyter
Project Jupyter evolved out of the IPython project (specifically the IPython notebook). The primary goal of Project Jupyter is to provide an interactive, web-browser driven, language-independent programming environment. Jupyter can be deployed on the farmshare servers to enable an accessible, powerful, and persistent computational platform.
At the end of this guide, the resulting jupyter notebook will support:
- an encrypted, password enabled, web-browser enabled programming environment
- persistence of the environment for seven days (maximum duration of Stanford Kerberos tickets
- file/data storage on the Stanford AFS servers with 5GB user quota and automatic backups
- shared file/data storage to Class Disk AFS Space
- simple switching to any of the Stanford FarmShare systems
- python3 support
Installation
Jupyter setup is best performed via the Linux console. This tutorial will be mostly step-by-step, but it general familiarity with the Linux console is helpful. Useful Linux console tutorials are available as a free book, online tutorial, or video series.
SSH is a protocol that enables an encrypted remote shell into another system. SSH is the primary way that Jupyter will be installed and accessed. SSH into one of the farmshare computers. The corn cluster (corn.stanford.edu) is a good choice. Windows does not come with an SSH client, but putty is a a free and lightweight SSH client for Windows. Mac and Linux have ssh clients accessible via the terminal.
Bind to Jupyter virtual environment
corn01:~> bash jane@corn01:~$ source /afs/ir.stanford.edu/group/bil/jupyter/bin/activate (jupyter)jane@corn01:~$
These two commands switch the shell to bash and then update the environment to use a pre-built Jupyter installation.
This Jupyter environment has the following packages installed:
bokeh ipython jupyter matplotlib numpy pandas scipy seaborn statsmodels sympy
Create a Jupyter virtual environment
Although farmshare has python installed, jupyter is not yet installed/supported. Since users do not have administrative access,
Ipython
Ipython provides both a command line and browser based (notebook) interfaces. The development is quite fast, so the distribution provided packages tend to lag behind enough it makes sense to setup a python environment which incorporates the latest updates.
building an Ipython notebook environment
Login to a corn and make yourself an Ipython environment in the non-AFS space
mkdir -p /farmshare/user_data/$USER/ipythontest cd /farmshare/user_data/$USER/ipythontest virtualenv --system-site-packages devbranch devbranch/bin/pip install --upgrade ipython devbranch/bin/pip install --upgrade tornado devbranch/bin/pip install jsonschema cp -p /farmshare/software/examples/ipython/audio.ipynb .
To run Ipython notebook environment, first start it up:
devbranch/bin/ipython notebook `%pylab inline` --no-browser
You should see output similar to following:
2013-12-03 21:39:50.435 [NotebookApp] Created profile dir: u'/afs/ir.stanford.edu/users/b/i/bishopj/.config/ipython/profile_default' 2013-12-03 21:39:50.465 [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js 2013-12-03 21:39:50.599 [NotebookApp] Serving notebooks from local directory: /srv/zfs01/user_data/bishopj/ipyhtontest 2013-12-03 21:39:50.599 [NotebookApp] The IPython Notebook is running at: http://127.0.0.1:8888/ 2013-12-03 21:39:50.599 [NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Setup an ssh tunnel from your desktop system to the same corn system you ran the notebook on in previous step. In my case port 8888 and corn02.
ssh -L 8888:localhost:8888 corn02
Now again from your desktop system open a web browser and navigate to 'localhost:8888'
Now click on Cluster and startup your cluster
Click back on the Notebook tab and you should see an audio notebook. click on this.
Now you can load up a notebook and "play"