User Guide
From FarmShare
(→Installed Software) |
|||
| Line 43: | Line 43: | ||
== Installed Software == | == Installed Software == | ||
| - | + | FarmShare systems run Ubuntu 16.04 LTS, and most software is sourced from standard repositories. Additional software, including licensed software, is organized using environment modules and can be accessed using the <code>module</code> command. Users can build and/or install their own software in their home directories, either manually, or using a local package manager. FarmShare supports running software packaged as Singularity containers. | |
== Running jobs on the cluster == | == Running jobs on the cluster == | ||
Revision as of 12:24, 28 August 2017
Contact SRCC staff for support at: srcc-support@stanford.edu, or post questions and concerns to the community discussion list at: farmshare-discuss@lists.stanford.edu.
Contents |
Connecting
Log into rice.stanford.edu. Authentication is by SUNet ID and password (or GSSAPI), and two-step authentication is required. A suggested configuration for OpenSSH and recommendations for two popular SSH clients for Windows can be found in Advanced Connection Options.
Storage
FarmShare is not approved for use with high-risk data, including protected health information and personally identifiable information.
Home
Home directories are served (via NFS 4) from a dedicated file server, and per-user quota is currently 48 GB. Users may exceed this soft limit for up to 7 days, up to a hard limit of 64 GB.
AFS
AFS is accessible from rice systems only. A link to each user's AFS home directory, ~/afs-home is provided as a convenience, but should only be used to access files in the legacy environment, and for transferring data. It should not be used as a working directory when submitting batch jobs, as AFS is not accessible from compute nodes. Please note that a valid Kerberos ticket and an AFS token is required to access locations in AFS; run kinit && aklog to re-authenticate if you have trouble accessing any AFS directory.
The default, per-user quota for AFS home directories is 5 GB, but you may have additional quota due to your enrollment in certain courses, and you can request additional quota (up to 25 GB total) with faculty sponsorship. AFS is backed up every night, and backups are kept for 30 days. The most recent snapshot of your AFS home directory is available in the .backup subdirectory, and you can request recovery from older backups by submitting a HelpSU ticket.
Scratch
Scratch storage is available in /farmshare/user_data, and each user is provided with a personal scratch directory, /farmshare/user_data/sunetid. The total volume size is currently 126 TB; quotas are not currently enforced, but old files may be purged without warning. The scratch volume is not backed up, and is not suitable for long-term storage, but can be used as working storage for batch jobs, and as a short-term staging area for data waiting to be archived to permanent storage.
Temp
Local /tmp storage is available on most nodes, but size varies from node to node. On rice systems, /tmp is 512 GB, with a per-user quota of 128 GB. Users may exceed this soft limit for up to 7 days, up to a hard limit of 192 GB, and space is regularly reclaimed from files older than 7 days.
File Transfer
Using SSH
FarmShare supports any file-transfer method using SSH as a transport, including standard tools like scp, sftp, and rsync on Linux and macOS systems, and SFTP clients like Fetch for macOS and SecureFX for Windows.
You can also use FUSE and SSHFS to mount your FarmShare home and scratch directories. Most Linux distributions provide a standard sshfs package. On macOS you can use Homebrew to install the sshfs package, or download FUSE and SSHFS installers from the FUSE for macOS project. Support for this option on Windows typically requires commercial software (like ExpanDrive).
Using AFS
You can use the native OpenAFS client to access files in AFS, including your AFS home directory. Most Linux distributions provide standard openafs packages. The University provides installers for the macOS and Windows clients.
You can also use WebAFS to transfer files to and from locations in AFS using a web browser.
Installed Software
FarmShare systems run Ubuntu 16.04 LTS, and most software is sourced from standard repositories. Additional software, including licensed software, is organized using environment modules and can be accessed using the module command. Users can build and/or install their own software in their home directories, either manually, or using a local package manager. FarmShare supports running software packaged as Singularity containers.
Running jobs on the cluster
We use Grid Engine (used to be Sun Grid Engine (SGE)). There are three types of jobs: interactive, batch and parallel. You can start by reading the man page for 'sge_intro'. Then the man page for 'qsub'. We currently have a limit of 3000 jobs (running and/or queued) per user. We don't currently allow interactive jobs on the barleys because you can run interactive tasks on the corns. Job scheduling uses simple fairshare (modified by resource requirements).
Make sure you have your kerberos credentials before submitting jobs or else they will not be able to access your files in AFS.
Running batch jobs
Use 'qsub'. This will allocate one slot on the cluster. See the bottom of the qsub man page for an example. Google 'SGE qsub' for more help.
Check how much memory your job uses. You can run just one job and see its peak memory usage after it's done. The standard barley node is 24 cores and 96GB RAM, so you shouldn't use more than 4GB/core. Make sure your submitted job doesn't use too much memory or it can crash the node.
Running array jobs
For jobs that vary only by one parameter, it is easier to submit an "array" job to reduce the amount of output in qstat. If you want to be a good citizen and you're submitting an array job with thousands of tasks, you may want to limit how many tasks you run simultaneously, using the -tc parameter to qsub.
Running parallel jobs
Use 'qsub' with the '-pe' parameter. Using the '-pe' parameter allows you to request more than one slot per job. We have several different "parallel environments" defined, they differ in how the slots are allocated. If you want your slots on the same node, use '-pe fah'. If you want your slots spread across nodes, use '-pe orte'. Use 'qconf -sp orte' to see the settings, and 'man sge_pe' for more info.
Running OpenMPI jobs
See OpenMPI, contact farmshare-discuss with any questions.
job duration
There's a 48 hour limit on jobs in the regular queue, 15 min in the test queue and 7 days in long queue. You can use '-l h_rt=xx:xx:xx' to tell the scheduler how long your job will run, and your job will be killed if it hits that time limit. Your job will make it into the long queue if and only if you request "-l longq=1". Your job will be killed (sent SIG_KILL) when you reach the h_rt limit that you set for yourself.
So the longest job that you can submit currently is 7 days, use "-l h_rt=168:00:00". But you should submit jobs less than 48hrs long, because there are many more regular job slots than long job slots.
When jobs fail, you typically have to re-run them. So try to split them into many small chunks (but not too many).
checking on your jobs
Use the qstat command to check on your currently pending/running jobs. Use the '-M' flag to qsub to have the system e-mail you about your job if you want. Look through your output files for output of the job stdout and stderr streams. Use the qacct command on machine senpai2 (because that's where the accounting file lives) to see some information about jobs that already finished, e.g. qacct -j JOBID. If there is no record of the job in qacct, that means it didn't get written to the accounting file, which means it failed in an unusual way. Look at your output files to see what the error was.
I usually look at the unfriendly output of this command:
qstat -f -u '*'
You can look at some slighly more friendly job status output. Try this script to see current memory usage per job:
/farmshare/user_data/chekh/qmem/qmem -u
Or this pie chart http://www.stanford.edu/~bishopj/farmsharemem/ (give it a minute or two to self-update)
