Sun Grid Engine

From VISTA LAB WIKI

Jump to: navigation, search

SGE stands for Sun Grid Engine, which is hosted on azure. The point of the SGE is to allow us to run code simultaneously on many different processors and computers, thus making it faster to complete your analyses. All non-trivial usage of the dedicated cluster machines (e.g. azure) should be routed through SGE so that load gets distributed fairly. In order to submit and check on jobs, you must log onto an execution host (i.e. a machine that accepts SGE jobs) such as azure or chroma. Information about SGE can be found at http://gridengine.sunsource.net. Manuals for SGE can be found here: Sun N1 GridEngine 6.1 Documentation.

Contents

[edit] Submitting jobs

  • Kendrick wrote sgerun.m and sgestat.m, MATLAB functions that make it easy to use SGE from within MATLAB. Please see the below section "How to start using the SGE" for more information.

Checking on your jobs:

  • To check the status of your jobs: qstat
  • To get detailed information about a job: qstat -j <jobnumber>
  • To check various SGE-related stuff using a GUI: qmon
  • To delete a job: qdel <jobnumber>
  • To check the current execution hosts: qconf -sel
  • To see the current submit hosts: qconf -ss
  • To check the status of the SGE hosts: qhost

[edit] Policies

  • Each individual job that you create should require a non-trivial amount of computational time (e.g. 10+ minutes). This is because there is overhead involved in creating and managing jobs, loading and saving data, and also in starting up MATLAB processes. On the other hand, each job should be relatively short (e.g. less than 6 hours) if possible. This is because the SGE can distribute the load fairly only when jobs are quickly cycled in and out (I think).
  • With regards to memory usage, we want to try to ensure that the jobs run by the SGE do not overload the memory usage on a given machine. Certain machines (e.g. azure) are set such that the maximum memory usage by SGE is the physical memory in the machine. Other machines (e.g. desktop machines) are set such that the maximum memory usage by SGE is 1-2 GB less than the physical memory in the machine, which allows memory to be used for other purposes (e.g. web browsing). The SGE is set up such that it will not put a job on a machine if that job will go over the limit set for that machine. To tell SGE about memory usage, each job is submitted with an estimate of memory usage. The default, which is used if the "-l virtual_free" flag is not specified, is 100M. To be a nice citizen, if you think that your job will take more than 100M of memory, you should use a flag like "-l virtual_free=2000M" when using the qsub command. Note that an input parameter that implements this feature is part of sgerun.m already.
  • If your desktop machine is an execution host, then you (the owner) will have the privilege to control SGE's usage of your machine. Controlling the SGE is easily accomplished using the qmon GUI utility. See section "Taking control of your machine" below.
  • If for whatever reason you find it necessary to kill jobs, reboot machines, etc., that is okay. The SGE is configured to re-run jobs in the sense that if it detects that any job has failed due to a crash, it will automatically reschedule the job for execution. This implies that you should make sure that it is safe to re-run any job that you submit to the SGE. (Actually, it is possible to specify that a job should not be re-run, but unless someone needs that functionality, let's just keep it simple.)

[edit] Notes

  • You should check to make sure that the same code gets run whenever you are logged onto any of the execution hosts. This means, for example, that the same MATLAB version should be used regardless of what host is used.
  • The SGE code itself includes many features which sgerun.m does not make use of. We can revise sgerun.m as the need arises...
  • Note that the SGE is general and can run any code, not just MATLAB code.
  • The SGE allows interactive jobs, which I think means that you can start a job that is as if you are logged onto some random machine. You could experiment with "qlogin -now no" and "qsh -now no" if you like.
  • Using the qmon utility, if you find a job of yours that is stuck on a suspended queue, you can either wait for that machine to be resumed again, or you can explicitly reschedule the job, which means that the job will be removed and then re-added to the queue.

[edit] Taking control of your machine

  • If you own an execution host and you find that the SGE is taking too much of your computer's resources, you have three options. The first option is to reduce the number of SGE jobs that are run on your machine. To do this, type:
 ~knk/bin/sgesetslot N

where N is the number of jobs (e.g. 0, 1, 2, 3, 4). This will cause the SGE to attempt to limit the number of jobs on your machine to N. Note that it may take a few minutes to equilibriate itself. Moreover, note that the SGE may exhibit some oscillatory behavior whereby the actual number of processes running might fluctuate. If you find that the results of the first option are unsatisfactory, a second, more drastic option is to simply suspend the SGE on your machine. To do this, type:

 ~knk/bin/sgeoff

This will suspend all SGE jobs on your machine. You can resume SGE jobs on your machine by calling:

 ~knk/bin/sgeon

This will cause the jobs that had been suspended to magically resume again. Note that suspended jobs still take up memory. If you start up processes that require substantial amounts of memory, there may be some overhead in swapping out the memory held by the suspended jobs. After an initial overhead period, your machine should behave nicely. If, for whatever reason, it doesn't, the third and most drastic option is to kill the processes on your machine (thus, losing the computations achieved up to this point). You can do this in a relatively nice way by rescheduling the processes that are on your machine (through the qmon utility). You can also do this by just killing the processes through the command-line (you will need sudo privileges to do this). Jobs that are rescheduled or killed will simply be added back the queue for execution at a later time.

  • Every night at 6pm and 11pm, we automatically reset the number of SGE jobs and resume the SGE on all machines (using a cron script under knk@chroma). The purpose of this is to make it such that you don't have to remember to reset things yourself. However, if you really want the SGE to stay off, you can type:
 ~knk/bin/sgeoffpermanent

This will suspend all SGE jobs and make it such that the nightly resume will not occur for your machine. To remove the permanent-off setting, issue the command again.

[edit] How to start using the SGE

Initial setup:

1. Clone the git repository "knkutils" and add it to your MATLAB path [NOTE: this repository assumes at least MATLAB 7.6]. (If you already have the repository, do an SVN update to ensure that you have the latest version!):

 cd ~
 svn checkout https://white.stanford.edu/repos/kendrick
 echo "addpath('~/kendrick');" >> ~/matlab/startup.m

2. Make the directory that will contain SGE output:

 mkdir ~/sgeoutput

3. Copy a sample matlabsge.sh script to your home directory:

 cp /home/knk/Public/matlabsge.sh ~/

4. Edit the script with the specific MATLAB call that you want:

 nano ~/matlabsge.sh

5. Ensure that the script is executable:

 chmod +x ~/matlabsge.sh

6. Add the SGE output directory to your MATLAB path:

 echo "addpath('~/sgeoutput');" >> ~/matlab/startup.m

7. Add this to your .bashrc shell file (necessary to set path and other SGE-related variables):

 if [ -f /azure/scr1/gridengine/default/common/settings.sh ]; then
   . /azure/scr1/gridengine/default/common/settings.sh
 fi

If you use csh, you can add this to your .cshrc shell file:

 if (-f /azure/scr1/gridengine/default/common/settings.csh) then
   source /azure/scr1/gridengine/default/common/settings.csh
 endif

Try it out:

1. Log onto azure, chroma, or some other machine that is an execution host:

 ssh azure

2. Start MATLAB:

 matlab

3. Submit a job called test1 that simply issues the 'ls' command:

 sgerun('ls','test1',0);

4. Monitor your job:

 !qmon &
   OR
 sgestat

5. When job is completed, inspect the results:

 !cat ~/sgeoutput/job_test1.o*

6. Submit a job called test2 that makes use of the <wantworkspace> functionality:

 somedata = [0 1 2];
 sgerun('somedata2 = somedata + 1','test2',1);

7. When job is completed, inspect the results:

 !cat ~/sgeoutput/job_test2.o*

8. Submit a job called test3 that farms out five different jobs and also saves output:

 sgerun([],'test3',0,1:5);
 result = jobindex.^2;
 save(sprintf('~/test%d.mat',jobindex));
 .

9. When job is completed, inspect the results:

 loadmulti('~/test*.mat','result',2)

10. Now you are ready to do anything!

[edit] System Administration notes

We are running GE 6.1 update 6. The master host is azure, using the sge-root directory /azure/scr1/gridengine.

Setup and installation

 # ensure a clean starting point (on all hosts)
 sudo /etc/init.d/sgemaster stop
 sudo rm -rf /usr/share/gridengine
 cd /usr/bin
 sudo rm -f qacct  qalter  qconf  qdel  qhold  qhost  qlogin  qmake  qmod  qmon  
 sudo rm -f qping  qquota  qresub  qrls  qrsh	qselect  qsh  qstat  qsub  qtcsh  
 sudo rm -f sge_coshepherd  sge_execd  sgepasswd	sge_qmaster  sge_schedd  sge_shadowd  sge_shepherd
 sudo rm -rf /usr/libexec/gridengine
 sudo rm -rf /usr/lib64/gridengine
 sudo rm -rf /etc/init.d/sge*
 
 # prepare installation files
 cd ge-6.1u6
 /bin/sh
 for i in *.gz ; do tar -zxvf $i ; done
 sudo mkdir /azure/scr1/gridengine
 sudo cp -r ge-6.1u6/* /azure/scr1/gridengine/
 
 # get the permissions ready
 sudo chmod -R 777 /azure/scr1/gridengine
 sudo chown -R sgeadmin:sgeadmin /azure/scr1/gridengine
 
 # install the master host
 cd /azure/scr1/gridengine
 sudo env SGE_ROOT=/azure/scr1/gridengine ./install_qmaster
   (install as sgeadmin)
   (do file permissions)
   (single DNS domain)
   (gid range 16000-16100)
   (kendrick@post.harvard.edu)
   (do not add hosts)
 
 # check the source thing suggested by the installer works
 source /azure/scr1/gridengine/default/common/settings.csh
 qconf -sh
 
 # add execution hosts on the master host
 cd /azure/scr1/gridengine
 sudo env SGE_ROOT=/azure/scr1/gridengine bin/lx24-amd64/qconf -ah chroma
 
 # on the execution host, install it
 cd /azure/scr1/gridengine
 sudo env SGE_ROOT=/azure/scr1/gridengine ./install_execd
 
 # on all hosts (who want to run qmon), need to install some additional stuff
 sudo rpm --force -ivh openmotif-2.3.3-1.fc10.x86_64.rpm  # well, choose the appropriate .rpm
 ls -la /usr/lib64/libX*
 sudo ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.3
 
 # check qmon works (through sudo) and do some tasks
 cd /azure/scr1/gridengine
 chmod 644 ~/.Xauthority
 sudo env SGE_ROOT=/azure/scr1/gridengine XAUTHORITY=/home/knk/.Xauthority bin/lx24-amd64/qmon
   (add knk,lmperry as a manager)
   (In "Complex Configuration", set virtual_free to be CONSUMABLE=YES, default=100M)
   (Modify all.q such that for @/, make the Suspend Interval 1 minute, make Rerun Jobs checked,
    and Job's Nice to +20)
   (In "Scheduler Configuration" -> "Load Adjustment", make a single entry with load_short set to 1)
   (In "Scheduler Configuration" -> "General Parameters", make Schedule Interval 15 s)
 
 # now, run qmon as knk (on chroma)
 qmon
 
 # add submit hosts
 qconf -as azure
 qconf -as chroma

Various checks and inspections

 # check daemons
 ps aux | grep sge
 
 # start the daemons (on the appropriate machines)
 sudo /azure/scr1/gridengine/default/common/sgemaster start
 sudo /azure/scr1/gridengine/default/common/sgeexecd start
 
 # various inspections
 qconf -sconf
 qhost
 qstat -f
 qstat -j 999
 
 # submit a job to test the queue
 qsub /azure/scr1/gridengine/examples/jobs/simple.sh
 
 # reset a queue:
 qmod -cq all.q@azure.Stanford.EDU
 
 # check log messages
 ls -lar /tmp/
 tail -f /tmp/execd*
 tail -f /tmp/qmaster*
 tail -f /azure/scr1/gridengine/default/spool/*/messages

Adding new execution hosts (example below uses chroma)

 # add execution host to the SGE daemon
 ssh azure
 cd /azure/scr1/gridengine
 sudo env SGE_ROOT=/azure/scr1/gridengine bin/lx24-amd64/qconf -ah chroma
 
 # do some checks on the execution host
 ssh chroma
   # the group should resolve to "sgeadmin"
 id sgeadmin
   # if not, then ensure an entry in the group file [HACK: the yp should already do this for us!]
 sudo nano /etc/group
   # next, ensure that the hosts file looks something like this (if necessary):
   # 127.0.0.1		localhost.localdomain localhost
   # ::1		localhost6.localdomain6 localhost6
   # 171.64.204.45		chroma.stanford.edu chroma
 sudo nano /etc/hosts
   # check if sge stuff is defined in services file
 grep sge /etc/services
   # if not, edit it to include the following:
   # sge_qmaster	6444/tcp		# Grid Engine Qmaster Service
   # sge_qmaster	6444/udp		# Grid Engine Qmaster Service
   # sge_execd	6445/tcp		# Grid Engine Execution Service
   # sge_execd	6445/udp		# Grid Engine Execution Service
 sudo nano /etc/services
 
 # install SGE on the execution host
 ssh chroma
 cd /azure/scr1/gridengine
 sudo env SGE_ROOT=/azure/scr1/gridengine ./install_execd
   (just press enter and use all defaults)
 
 # see if qmon runs on the execution host
 qmon
 
 # if not, you may need to install some additional stuff
 cd /azure/scr1/gridengine
 cat /etc/fedora-release
 sudo rpm --force -ivh openmotif/openmotif-2.3.3-1.fc12.x86_64.rpm  # well, choose the appropriate .rpm
 sudo rpm --force -ivh openmotif/openmotif-2.3.3-1.fc10.x86_64.rpm  # well, choose the appropriate .rpm
 sudo rpm --force -ivh openmotif/openmotif-2.3.3-1.fc8.x86_64.rpm   # well, choose the appropriate .rpm
 ls -la /usr/lib64/libX*
 sudo ln -s /usr/lib64/libXm.so.4 /usr/lib64/libXm.so.3
 
 # run qmon and do some configuration
 qmon
   (In "Queue Control", modify the all.q queue and set the new machine's owner 
   (e.g. knk), the number of slots if necessary (e.g. 4 for a 4-processor machine).
   In "Host Configuration", modify the new execution host to have a consumable attribute 
   'virtual_free' that is set to be either the physical memory limit (e.g. 8000000k) 
   or something a little less to allow for web browsing, etc. (e.g. 6000000k).)
   (add the owner to become a manager!)
   (also, set up mem_used on the thresholds 6G /7G (or whatever))
 
 # add submit hosts
 qconf -as chroma

Scripts, etc.

 # please see ~knk/bin as well as the crontab for knk@chroma
Personal tools