Jupyter
From FarmShare
(→Bind to Jupyter virtual environment and install default configuration: rename wrapper script) |
(→Technical Details: created new section) |
||
Line 250: | Line 250: | ||
[[File:jupyter_julia_demo.png]] | [[File:jupyter_julia_demo.png]] | ||
+ | |||
+ | == Installing custom Jupyter notebook == | ||
+ | |||
+ | |||
== Technical Details == | == Technical Details == |
Revision as of 12:37, 13 April 2017
Contents |
Introduction
Project Jupyter evolved out of the IPython project (specifically the IPython notebook) with the goal to provide an interactive, web-browser driven, language-independent programming environment. Jupyter notebooks can be deployed on the FarmShare servers to enable an accessible, powerful, and persistent computational platform.
Features
At the end of this guide, the resulting Jupyter notebook will support:
- Python, MATLAB, R, SAS, and Julia programming languages
- an encrypted, token protected, and web-browser enabled programming environment
- indefinite persistence of the Jupyter notebook server environment with simple weekly renewals (the maximum duration of Stanford Kerberos tickets)
- file/data storage on the Stanford AFS servers (5GB user quota, Stanford-wide, automatic backups)
- shared file/data storage to Class Disk AFS Space
- shared file/data storage to Group AFS Space
- easy deployment on any of the Stanford FarmShare systems or any Linux system integrated with Stanford Kerberos and AFS.
Overview
The guide consists of three sections: installation, Jupyter server, and client connection.
- Installation is only performed once per user.
- The Jupyter server needs to be started once per FarmShare server used. It will continue running until shut down (or the server is restarted).
- Client connection must happen every time a client computer reconnects to the Internet (e.g., a laptop wakes from sleep).
Commands to be typed in are in bold, text to substitute is in red, and the corn server and TCP port to note are in blue.
Installation
This only needs to be performed once per user.
In a terminal, SSH into a FarmShare computer.
$ ssh jane@corn.stanford.edu
substituting SUNet ID for jane.
A terminal application and SSH client are shipped with Mac and Linux systems. Windows does not come with an SSH client, but putty is a a free and lightweight SSH client for Windows.
Bind to Jupyter virtual environment and install default configuration
corn99:~> bash (ignore this command if prompt already looks like: jane@corn99:~$ ) jane@corn99:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate (jupyter)jane@corn99:~$ generate_jupyter_wrapper
Jupyter server
This only needs to be performed once per FarmShare server used (and after every server reboot).
SSH info FarmShare if already not already in a system (ok to continue on from existing SSH session used for Installation).
Launch Jupyter server
Make a note of exactly which FarmShare server is being used (e.g., corn14, corn22, etc).
(jupyter)jane@corn99:~$ pagsh (jupyter)jane@corn99:~$ kinit; aklog Password for jane@stanford.edu: (jupyter)jane@corn99:~$ tmux (new blank terminal appears) jane@corn99:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate (jupyter)jane@corn99:~$ keep_kerberos_afs Run: export KRB5CCNAME=FILE:/tmp/.krb5_jane.tgt jane@corn99:~$ export KRB5CCNAME=FILE:/tmp/.krb5_jane.tgt (jupyter)jane@corn99:~$ jupyter_start
Substitute SUNet ID for jane in the export
line (as instructed by the keep_kerberos_afs
script). If successful, the output of this command will be
IMPORTANT: This Jupyter notebook is listening on TCP port 9876 [I 17:23:59.661 NotebookApp] Loading IPython parallel extension [I 17:23:59.668 NotebookApp] Serving notebooks from local directory: /afs/ir.stanford.edu/users/j/a/jane [I 17:23:59.668 NotebookApp] 0 active kernels [I 17:23:59.668 NotebookApp] The Jupyter Notebook is running at: https://localhost:9876/?token=ba682763f27d8e2d59862badef28b0eaecb552529933176e [I 17:23:59.668 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [C 17:23:59.671 NotebookApp] Copy/paste this URL into your browser when you connect for the first time, to login with a token: https://localhost:9876/?token=ba682763f27d8e2d59862badef28b0eaecb552529933176e
The Jupyter notebook will not work until a client connection is established. Copy the https://localhost:9876/?token=...
URL and save it for subsequent use. Also note the TCP port in blue that the Jupyter server is listening on.
Confirm Jupyter server status
Confirm that the Jupyter server was successfully launched.
Control-B then C (releasing all keys after the Control-B before hitting C) (opens a new tmux window and switches to it) jane@corn99:/afs/ir.stanford.edu/users/j/a/jane$ cd jane@corn99:~$ source /afs/ir.stanford.edu/group/bil/env/jupyter/bin/activate (jupyter)jane@corn99:~$ jupyter notebook list Currently running servers: https://localhost:9876/?token=ba682763f27d8e2d59862badef28b0eaecb552529933176e :: /afs/ir.stanford.edu/users/j/a/jane
If successful, the output will list the new running Jupyter notebook, its TCP port, token, and home directory where the notebook was launched.
Detach and logout
Detach from tmux and logout of FarmShare.
Control-B then D (releasing all keys after the Control-B before hitting D) (detaches tmux) (jupyter)jane@corn99:~$ exit jane@corn99:~$ exit jane@corn99:~$ exit (if exit does not work here, use logout)
Client connection
This is the only step that needs to be performed every time (e.g., a client laptop wakes from sleep, connecting from a different laptop) once the installation is complete and the Jupyter server is running.
SSH into the same FarmShare system that the Jupyter server was started in above, tunneling the TCP port that the Jupyter server is listening on.
$ ssh jane@corn99.stanford.edu -L 9876:localhost:9876
substituting in the appropriate SUNet ID for jane
, the appropriate FarmShare server hostname for corn99
, and the appropriate TCP port for 9876
. Windows users can setup an SSH local tunnel using menu options in putty.
Once logged in, paste the https://localhost:9876/?token=...
URL provided from the jupyter_start
script into a web browser to connect to the Jupyter notebook. Ignore any browser security warnings.
Note: Jupyter works best with Firefox or Chrome browsers. Safari and Internet Explorer are not well supported.
Shut down Jupyter
It's best to shut down Jupyter to free resources for others when finished and will not be used for an extended period (e.g., greater than two weeks).
SSH into the same FarmShare system that the Jupyter server was started in.
$ ssh jane@corn99.stanford.edu
substituting in the appropriate SUNet ID for jane
and the appropriate FarmShare server hostname for corn99
.
Return to tmux window 0 where Jupyter was launched, shut down the server, and exit tmux.
corn99:~> tmux attach -d Control-B then 0 (releasing all keys after the Control-B before hitting 0) (returns to tmux window 0 with lots Jupyter notebook output) [I 17:23:59.671 NotebookApp] Control-C then Control-C (releasing all keys after the Control-C before hitting Control-C a second time)
This will shut down the Jupyter notebook. Log out of FarmShare.
Renewing virtual terminal
This only needs to be done when the Kerberos credentials expire after a week and the notebook no longer functions. To restore the notebook after this time, SSH into the same FarmShare system used to create the virtual terminal. Renewal can take place at any time after a week when use is desired.
$ ssh jane@corn99.stanford.edu
substituting SUNet ID for jane and the appropriate FarmShare server for corn99.
corn99:~> tmux attach -d (jupyter)jane@corn99:~$ kinit; aklog Password for jane@stanford.edu: (jupyter)jane@corn99:~$ keep_kerberos_afs Run: export KRB5CCNAME=FILE:/tmp/.krb5_jane.tgt Control-B then D (releasing all keys after the Control-B before hitting D) (detaches tmux)
This will extend the terminal's credentials for another week. The same notebook URL can be used without interruption. Logout of FarmShare.
Examples
Python 3
import numpy as np import plotly.offline as py import plotly.graph_objs as go py.init_notebook_mode(connected=True) x = np.r_[ -np.pi : np.pi : 0.1 ] py.iplot([go.Scatter(x=x, y=np.sin(x)), go.Scatter(x=x, y=np.cos(x))])
MATLAB
farmshare_plotly_init; x = -pi : 0.1 : pi; hold on; plot(x, sin(x)); plot(x, cos(x));
Note: If Jupyter installation was performed prior to 2017/04, run the following script in an ssh terminal to configure plotly for matlab: plotly_create_config
R
x = seq(-pi, pi, 0.1) plot(x, sin(x), type='l', lwd=4, col='blue', ann=FALSE) lines(x, cos(x), type='l', lwd=4, col='red')
SAS
data curves; do x = -constant("pi") to constant("pi") by 0.1; y = sin(x); z = cos(x); output; end; run; symbol1 interpol=join color=blue width=5; symbol2 interpol=join color=red width=5; axis1 label=none minor=none; proc gplot data=curves; plot y*x=1 z*x=2 / overlay vaxis=axis1 haxis=axis1; run;
Julia
using Plots plotly() plot([sin, cos], -pi, pi)
Installing custom Jupyter notebook
Technical Details
- Jupyter setup is best performed via the Linux console. This guide is mostly step-by-step, but general familiarity with Linux is helpful.
- Useful Linux tutorials are available as a free book, online tutorial, or video series.
- SSH provides an encrypted remote shell into another system and is the primary way that Jupyter will be installed and accessed.
- The first two commands of the installation (
bash
andsource
) switch the shell to bash and then update the environment to use a pre-built Jupyter installation. The prompt changes to have the(jupyter)
prefix when the environment change is successful. SUNet IDs ceated after 2014 have their default shell set to bash and can omit the call tobash
.
- If of interest, the pip packages installed for the Jupyter environment are stored in a
requirements.txt
file at/afs/ir.stanford.edu/group/bil/env/jupyter/requirements.txt
. Additional packages may be installed upon request.
- The
jupyter_config_wrapper
command calls a script that will create encryption keys for the Jupyter notebook, configure the jupyter config file, and sets the default tmux shell to bash. Note, this script modifies existing Jupyter notebook config files (if they exist), but will not overwrite any parameters that have changed from their default values.
- These instructions runs the notebook within a persistent virtual terminal so it will continue to run even after the user has logged out of FarmShare.
- The
jupyter_start
script accepts an optional port argument, specifying which port to connect to (e.g.,jupyter_start 9876
).
- Once the Jupyter notebook server is running, it is ready to accept client connections. For security, it only accepts connections from localhost (i.e., connections originating from that specific FarmShare system itself). Connections from other systems (e.g., a laptop) are created through SSH tunnels.
- Since the encryption keys were self-signed, the browser will warn about an insecure connection, but this warning can be disregarded.
- If this notebook URL is misplaced,
tmux attach -d
brings up the virtual terminal and the notebook status can be queried using the verification step on tmux window 1 usingjupyter notebook list
.