From FarmShare

Revision as of 01:07, 24 December 2016

Introduction

Project Jupyter evolved out of the IPython project (specifically the IPython notebook). The primary goal of Project Jupyter is to provide an interactive, web-browser driven, language-independent programming environment. Jupyter can be deployed on the farmshare servers to enable an accessible, powerful, and persistent computational platform.

FarmShare Specific Features

At the end of this guide, the resulting jupyter notebook will support:

• an encrypted, token protected, web-browser enabled programming environment
• persistence of the environment for seven days (the maximum duration of Stanford Kerberos tickets)
• file/data storage on the Stanford AFS servers with 5GB user quota and automatic backups
• shared file/data storage to Class Disk AFS Space
• simple switching to any of the Stanford FarmShare systems
• python3 default programming language

Installation

Jupyter setup is best performed via the Linux console. This guide will be mostly step-by-step, but general familiarity with Linux is helpful. Useful tutorials are available as a free book, online tutorial, or video series.

SSH into FarmShare

SSH is a protocol that enables an encrypted remote shell into another system. SSH is the primary way that Jupyter will be installed and accessed. SSH into one of the farmshare computers. The corn cluster (i.e., corn.stanford.edu) is a good choice. Windows does not come with an SSH client, but putty is a a free and lightweight SSH client for Windows. Mac and Linux have ssh clients accessible via the terminal.

Bind to Jupyter virtual environment

corn01:~> bash
jane@corn01:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate
(jupyter)jane@corn01:~$

These two commands switch the shell to bash and then update the environment to use a pre-built Jupyter installation. The prompt should have the (jupyter) prefix if the environment change is successful.

This Jupyter environment has the following packages installed:

bokeh ipython jupyter jupyter_contrib_nbextensions matplotlib numpy pandas scipy seaborn statsmodels sympy

Setup encryption and default configuration

(jupyter)jane@corn01:~$ /afs/ir.stanford.edu/group/bil/env/jupyter/scripts/jupyter_config_wrapper

This command calls a script that will create encryption keys for the Jupyter notebook and configure the jupyter config file to use these newly created keys. Note, this will modify existing Jupyter notebook config files (if they exist), but will not overwrite any parameters that have changed from their default values.

Installation is complete after this step is successful. Log out of FarmShare.

Alternative: Create personal Jupyter virtual environment

If the pre-built Jupyter environment is not sufficient (e.g., additional packages or different versions are needed), then a personal virtual environment can also be setup:

corn01:~> bash
jane@corn01:~$ virtualenv -p python3 jupyter_env
jane@corn01:~$ source jupyter_env/bin/activate
(jupyter_env)jane@corn01:~$ pip3 install jupyter <other desired packages>

Execution

Jupyter will be run within a persistent virtual terminal leveraging the keeptokens script so it can stay running even after the user has logged out of FarmShare.

Create virtual terminal

This only needs to be done once per FarmShare machine (and after every reboot).

SSH into a FarmShare system (e.g. corn). Make a note of exactly which corn server entered (e.g., corn14, corn22, etc).

corn99:~> pagsh
sh-4.3$ kinit -r 7d; aklog
Password for jane@stanford.edu:
sh-4.3$ bash
jane@corn99:~$ tmux
jane@corn99:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate
(jupyter)jane@corn99:~$ keeptoken
(jupyter)jane@corn99:~$ source /tmp/.krbhold_jane.sh
(jupyter)jane@corn99:~$ /afs/ir.stanford.edu/group/bil/env/scripts/jupyter_start

Substitute sunetid for jane in the second source command as instructed by keeptoken. Copy the https://localhos... URL and save it for subsequent use. Also note the TCP port that the Jupyter notebook is listening on. Use Ctrl+b then d to detach from tmux. Logout of FarmShare.

Create SSH tunnel

The Jupyter notebook is now running and ready to accept connections. However, it only accepts connections from localhost (i.e., connections originating from the FarmShare system itself). To connect from another machine, an SSH tunnel can be used. SSH into the same FarmShare system that the virtual terminal was created on (e.g., corn14, corn22, etc) by using the server's FQDN (e.g., corn14.stanford.edu) with a local port tunnel to the TCP port that the server is listening on.

In Mac/Linux, setup the SSH tunnel in a terminal:

$ ssh jane@corn99.stanford.edu -L 9999:localhost:9999

substituting in the appropriate sunetid for jane, the appropriate FarmShare server hostname for corn99, and the appropriate TCP port for 9999. Windows users can setup an SSH local tunnel using menu options in putty

Once the ssh tunnel has been established, paste the https://localhos... URL provided from the Jupyter notebook launch into a web browser to connect to the notebook home page.

Renewing virtual terminal

The virtual terminal created by the above instructions will persist on the FarmShare server for seven days, after which its Kerberos tickets will expire and the Jupyter notebook will no longer function. To restore the notebook after this time, ssh into the same FarmShare system used to create the virtual terminal.

corn99:~> bash
(jupyter)jane@corn99:~$ tmux attach -d
Ctrl+c then y (to kill the defunct but running Jupyter notebook)
(jupyter)jane@corn99:~$ kinit -r 7d; aklog
Password for jane@stanford.edu:
(jupyter)jane@corn99:~$ /afs/ir.stanford.edu/group/bil/env/scripts/jupyter_start

Again, copy the https://localhos... URL, as this replaces the prior URL for the earlier Jupyter notebook session.