SCIEN Software

From VISTA LAB WIKI

Revision as of 20:52, 21 November 2015 by Wandell (Talk | contribs)
Jump to: navigation, search

We are part of Stanford's Center for Image Systems Engineering. The software we develop for this work is distributed through our github repositories. These are links to those repositories

We also maintain some additional github repositories that are described in specific portions of the documentation.

Here is a local page describing the basic GIT operations.

Contents

Scientific methods

Software revisions

Data

We are storing critical data sets for ISETBIO and SCIEN on this Archiva repository on Black. BW to fill in details here.

PDCDATA

Needs updating. We will probably start maintaining these data sets elsewhere.

We maintain a repository for data files, PDCDATA. The data sets can be checked out from the ee svn repository in the usual way, just substituting pdcdata for pdcsoft. There exist folders for reflectances, illuminants, RGB images, multispectral scenes, sensors, etc.

If you wish to add data to this directory, please follow the guidelines outlined in the README files in the directories. All data of a particular type should have a consistent format (outlined in the README files) to ensure interoperability of code and data.

There is a separate repository for multispectral data for scenes because these files tend to be very large. The repository is called "scenedata" (https://white.stanford.edu/ee/scenedata).

Project on scientific transparency (POST)

Running Matlab Remotely

If you have an account on white, you can run Matlab remotely on other computers in the lab. This is a rather simple process.

Cygwin

If you are on Windows, you can use Cygwin/X [1] to have a graphical interface with Matlab running on a server. When installing the software, make sure to choose to install X as described here [2] and OpenSSH (under net category). Once installed and running XWin Server on your machine, you can login to a machine such as peach by [ssh -X2 username@peach].

Xming

If you are on a Windows machine, you can also use Xming to open xterminals [3] on the linux machines and run Matlab remotely.

From off-campus do this

1. Download and install Xming [4]
2. Use SecureCRT to login to white.stanford.edu
3. Type "ssh azure" to open a window on azure
4. Type "xterm &" to open an X window
5. Type  "matlabr2010aREAL &" to open Matlab in a separate window.

From on-campus you can log into azure or viridian or other machines directly, without logging into white first.

No graphics

If you don't need the graphical interface, you have another option. You need an SSH client such as SecureCRT if you are on Windows.

  1. Start your client
  2. SSH into one of the machines, e.g., azure.stanford.edu
  3. Type:  'matlab' and this will open up a version of Matlab in text only mode.

If you want to run a Matlab process that will continue in the background even if you close your SSH session. Execute the following command:

 nohup matlab < to_do.m > results.txt &

Here Matlab will execute the file to_do.m in the background and will save any text displayed in the Command Window to results.txt. The above command will also show also return the ID number for this task. If you execute the command 'top' you can see all the processes currently running on the machine. You can use this to check if your program is still running. If for some reason you would like to terminate your program, execute 'kill #####' where the number is the ID number of the process you want to kill. You can execute the following command to view the contents of results.txt:

 less results.txt

If you need to execute a file that is not in your home directory, you can change directories using:

 cd directory_name

This method will result in only the current folder being on Matlab's path. If you want to add all of the subdirectories of the current folder to Matlab's path, add the following command to your .m file:

 addpath(genpath(pwd))

The 'azure' computer can run multiple scripts at the same time. You can view the currently running processes by executing the command 'top'.

In order to transfer files to or from white, use a SFTP client such as WinSCP if you are on Windows and connect to white.stanford.edu.

Virtual Box

Andy Lin has us running Ubuntu on Windows machines using VirtualBox. This is a free virtual machine system from Oracle.

Install Virtual Box on a Windows machine. Then, download a version of Ubuntu.

You can then create a virtual instance of Ubuntu that will run on the Windows machine. Some suggestions:

  1. Set up the Machine properties for more than 1 processor.
  2. Set an execution cap that is less than 100 percent so the Windows side of your machine can continue to run
  3. Use more RAM (e.g., 4 or 5 GB).
  4. Use a virtual disk of 15-20GB at least.

We will try to make the notes here a little better as more people go through the process. Though it may be that only Andy and Brian have Windows machines.

Writing Matlab Files

When you are writing Matlab scripts or functions, please follow some guidelines in order to allow others to be able to use and understand your code and its outputs. It is important that the code is written in a readable modular fashion so that other people in the lab (now and in the future) may benefit from and work with your code.

At the top of each of your files, please add a header describing what it does. The suggested format for a simple example function appears below:

 function ycbcr = myrgb2ycbcr(rgb)
 % Converts an image using the color conversion used in JPEG
 % 
 %     ycbcr=myrgb2ycbcr(rgb)
 %
 % A few sentences about the function there ...
 %
 % ...
 % 
 %
 %INPUT:
 %   rgb:        Double array giving an RGB image with values between 0 and 255
 %               size(rgb)=m x n x 3
 %
 %OUTPUT:
 %   ycbcr:      Double array giving a YCbCr image with values between 0 and 255
 %               size(ycbcr)=m x n x 3
 %
 % See also:  Related functions
 %
 % Examples:
 %    Examples go here
 %
 %
 % (c) Copyright notice


Additional comments or notes can be added after the description of the output(s).


ISET

We use ISET for many projects. The ISET page contains instructions about downloading, distributing and using this tool.

SVN (TO BE DEPRECATED)

The SVN web-page in the vista-wiki has more complete documentation than this one. Here we add some specific notes about the SVN and software for the pdc projects.

All the Matlab code was stored in the subversion repository. You may wish to review how to use SVN to download projects from the repository into your local machine (sandbox).

The PDC SVN repository is stored on white (as part of the ee respository). The ee repository is divided into four sections iset-3.0, iset-4.0, pdcprojects and pdcsoft.

  • PDCSOFT: Contains PDC utilities
  • PDCPROJECTS: Contains specific projects
  • PDCDATA: Contains data sets used in digital imaging simulation
  • ISET-4.0: Image Systems Engineering Toolkit

These repositories are password protected for SVN download. To obtain a password, you must ask Brian Wandell.

We suggest you checkout the directories in the repository to a directory on your local machine called SVN . The methods for Linux/Mac and Windows differ somewhat (see below).

In the example, we assume you want to checkout the entire set of pdcprojects directories. It is also possible to check out only one of these folders, or the pdcsoft or the iset-4.0 directory.

When checking out a directory, you must fill in the project directory and the local directory. The images here are just examples. The main directories in the repository for PDC are accessed via https on white.stanford.edu/ee. Current sections are

  • iset-4.0
  • pdcprojects
  • pdcsoft
  • pdcdata

Checkout: Windows

On PCs we use TortoiseSVN

Checkout: Linux/MAC

Most Linux and Apple systems have SVN installed and a command line interface is typically used.

  • Make a local SVN directory
  • cd into that directory
  • To check out the entire repository use the command line
svn checkout https://white.stanford.edu/ee/pdcprojects
svn checkout https://white.stanford.edu/ee/pdcsoft
svn checkout https://white.stanford.edu/ee/iset-4.0
  • If you just want a portion of the very large repository, you can use
svn checkout https://white.stanford.edu/ee/pdcprojects/L3pipeline
svn checkout https://white.stanford.edu/ee/pdcprojects/LEDms

Checkout: Windows

PDC Checkout

We suggest usingTortoiseSVN - you'll need to restart after the install. You can download TortoiseSVN from here.

After the installation, check out the pdcprojects directory by

  • Right click for the context menu
  • Select the SVN Checkout option
  • This will bring up a window that looks like the one at the right

You should fill in the relevant repository (or just the specific directory within the respository). You can specify where you want the folder or repository to be located on your local machine.

There are many TortoiseSVN tools that make it easy to resolve conflicts and check for updates and commits.

Repository Browsing

To have an overview of the main PDC and ISET repositories, you can use the browsing feature of the SVN repositories.

These links show you the repository directories as well as the history of the different files and directories in the repositories.

Personal tools