SCIEN Software

From VISTA LAB WIKI

(Difference between revisions)
Jump to: navigation, search
(Project on scientific transparency (POST))
 
(10 intermediate revisions not shown)
Line 1: Line 1:
-
We are part of [http://scien.stanford.edu Stanford's Center for Image Systems Engineering].  The software we develop for this work is distributed through our [http://github.com github] repositories. These are links to those repositories
+
The software we develop for this work is distributed through our [http://github.com github] repositories.  
-
 
+
-
* [https://github.com/scienstanford SCIENSTANFORD]
+
-
* [https://github.com/isetbio ISETBIO]
+
-
 
+
-
We also maintain some additional github repositories that are described in specific portions of the documentation.
+
Here is a local page [http://white.stanford.edu/newlm/index.php/GIT describing the basic GIT operations].
Here is a local page [http://white.stanford.edu/newlm/index.php/GIT describing the basic GIT operations].
-
= Scientific methods =
+
= Software =
-
== Software revisions ==
+
We are part of [http://scien.stanford.edu Stanford's Center for Image Systems Engineering].  Several projects related to image systems engineering are stored at these accounts.
-
== Data ==
+
* [https://github.com/scienstanford SCIENSTANFORD]
 +
* [https://github.com/isetbio ISETBIO]
 +
* [http://imageval.com ImageVal Web site for ISET]
-
We are storing critical data sets for ISETBIO and SCIEN on this Archiva repository on Black.  BW to fill in details here.
 
-
== [[PDCDATA]] ==
+
The github repositories maintain a series of programs related to our database and computational work in our  [http://post.stanford.edu project on scientific transparency].
-
Needs updating. We will probably start maintaining these data sets elsewhere.
+
* [https://github.com/orgs/scitran/dashboard SCITRAN]
 +
* [https://github.com/scitran/core/wiki SCITRAN WIKI]
-
We maintain a repository for data files, [[PDCDATA]].  The data sets can be checked out from the ee svn repository in the usual way, just substituting pdcdata for pdcsoft.  There exist folders for reflectances, illuminants, RGB images, multispectral scenes, sensors, etc.
+
See also:
-
If you wish to add data to this directory, please follow the guidelines outlined in the README files in the directories.  All data of a particular type should have a consistent format (outlined in the README files) to ensure interoperability of code and data.
+
* [https://github.com/flywheel-io Flywheel Exchange, LLC]
 +
* [https://flywheel.io Flywheel Exchange, LLC]
-
There is a separate repository for multispectral data for scenes because these files tend to be very large.  The repository is called "scenedata"  (https://white.stanford.edu/ee/scenedata).
+
= Data =
-
== Project on scientific transparency (POST) ==
+
We are storing critical data sets for
-
= Running Matlab Remotely =
+
* [http://52.32.77.154/#browse~isetbio ISETBIO]
 +
* [http://52.32.77.154/#browse~scien SCIEN]
 +
* [http://52.32.77.154/#browse~vistasoft  VISTASOFT]
-
If you have an account on white, you can run Matlab remotely on other computers in the lab. This is a rather simple process.  
+
on the Archiva server.  We interact with these databases using the [https://github.com/isetbio/RemoteDataToolbox RemoteDataToolbox].
-
== Cygwin ==
 
-
If you are on Windows, you can use Cygwin/X [http://x.cygwin.com/] to have a graphical interface with Matlab running on a server.  When installing the software, make sure to choose to install X as described here [http://x.cygwin.com/docs/ug/setup.html] and OpenSSH (under net category).  Once installed and running XWin Server on your machine, you can login to a machine such as peach by [ssh -X2 username@peach].
 
-
== Xming ==
 
-
If you are on a Windows machine, you can also use Xming to open xterminals [http://www.straightrunning.com/XmingNotes] on the linux machines and run Matlab remotely.
+
= Running Matlab Remotely =
-
From off-campus do this
+
We use Microsoft Remote Desktop to login to servers and run remotely.
-
 
+
-
1. Download and install Xming [http://sourceforge.net/projects/xming/files/Xming/6.9.0.31/Xming-6-9-0-31-setup.exe/download]
+
-
2. Use SecureCRT to login to white.stanford.edu
+
-
3. Type "ssh azure" to open a window on azure
+
-
4. Type "xterm &" to open an X window
+
-
5. Type  "matlabr2010aREAL &" to open Matlab in a separate window.
+
-
 
+
-
From on-campus you can log into azure or viridian or other machines directly, without logging into white first.
+
-
 
+
-
== No graphics ==
+
-
 
+
-
If you don't need the graphical interface, you have another option.  You need an SSH client such as [http://www.stanford.edu/services/ess/pc/securecrt.html SecureCRT] if you are on Windows.
+
-
 
+
-
  1. Start your client
+
-
  2. SSH into one of the machines, e.g., azure.stanford.edu
+
-
  3. Type:  'matlab' and this will open up a version of Matlab in text only mode.
+
-
 
+
-
If you want to run a Matlab process that will continue in the background even if you close your SSH session.  Execute the following command:
+
-
 
+
-
  nohup matlab < to_do.m > results.txt &
+
-
 
+
-
Here Matlab will execute the file to_do.m in the background and will save any text displayed in the Command Window to results.txt.  The above command will also show also return the ID number for this task.  If you execute the command 'top' you can see all the processes currently running on the machine.  You can use this to check if your program is still running.  If for some reason you would like to terminate your program, execute 'kill #####' where the number is the ID number of the process you want to kill.  You can execute the following command to view the contents of results.txt:
+
-
 
+
-
  less results.txt
+
-
 
+
-
If you need to execute a file that is not in your home directory, you can change directories using:
+
-
 
+
-
  cd directory_name
+
-
 
+
-
This method will result in only the current folder being on Matlab's path.  If you want to add all of the subdirectories of the current folder to Matlab's path, add the following command to your .m file:
+
-
 
+
-
  addpath(genpath(pwd))
+
-
 
+
-
The 'azure' computer can run multiple scripts at the same time.  You can view the currently running processes by executing the command 'top'.
+
-
 
+
-
In order to transfer files to or from white, use a SFTP client such as [http://winscp.net/eng/index.php WinSCP] if you are on Windows and connect to white.stanford.edu.
+
= Virtual Box =
= Virtual Box =
Line 129: Line 90:
Additional comments or notes can be added after the description of the output(s).
Additional comments or notes can be added after the description of the output(s).
 +
= [http://white.stanford.edu/newlm/index.php/SVN SVN] (DEPRECATED) =
-
= [[ISET]] =  
+
== Repository Browsing ==
-
We use [[ISET]] for many projects.  The [[ISET]] page contains instructions about downloading, distributing and using this tool.
+
As far as I know, everything is now on github or the RDT (Maven/Archiva) repository.
-
= [http://white.stanford.edu/newlm/index.php/SVN SVN] (TO BE DEPRECATED) =
+
I leave these links here just in case we forgot something.
-
 
+
-
The [http://white.stanford.edu/newlm/index.php/SVN SVN web-page in the vista-wiki] has more complete documentation than this one. Here we add some specific notes about the SVN and software for the pdc projects.
+
-
 
+
-
All the Matlab  code was stored in the subversion repository. You may wish to review how to use [http://white.stanford.edu/newlm/index.php/SVN SVN ] to download projects from the repository into your local machine (sandbox).
+
-
 
+
-
The PDC SVN repository is stored on white (as part of the ee respository).  The ee repository is divided into four sections iset-3.0, iset-4.0, pdcprojects and pdcsoft. 
+
-
 
+
-
* PDCSOFT:      Contains PDC utilities
+
-
* PDCPROJECTS:  Contains specific projects
+
-
* PDCDATA:      Contains data sets used in digital imaging simulation
+
-
* ISET-4.0:    Image Systems Engineering Toolkit
+
-
 
+
-
These repositories are password protected for SVN download. To obtain a password, you must ask Brian Wandell.
+
-
 
+
-
We suggest you checkout the directories in the repository to a directory on your local machine called <b> SVN </b>. The methods for Linux/Mac and Windows differ somewhat (see below). 
+
-
 
+
-
In the example, we assume you want to checkout the entire set of  <b> pdcprojects </b> directories. It is also possible to check out only one of these folders, or the <b> pdcsoft </b> or the <b> iset-4.0 </b> directory.
+
-
 
+
-
When checking out a directory, you must fill in the project directory and the local directory. The images here are just examples. The main directories in the repository for PDC are accessed via https on white.stanford.edu/ee.  Current sections are
+
-
 
+
-
* iset-4.0
+
-
* pdcprojects
+
-
* pdcsoft
+
-
* pdcdata
+
-
 
+
-
== Checkout: Windows ==
+
-
 
+
-
On PCs we use [http://tortoisesvn.net/ TortoiseSVN]
+
-
 
+
-
== Checkout: Linux/MAC ==
+
-
Most Linux and Apple systems have SVN installed and a command line interface is typically used.
+
-
 
+
-
* Make a local SVN directory
+
-
* cd into that directory
+
-
* To check out the entire repository use the command line
+
-
: svn checkout <nowiki> https://white.stanford.edu/ee/pdcprojects </nowiki> <br>
+
-
: svn checkout <nowiki> https://white.stanford.edu/ee/pdcsoft </nowiki> <br>
+
-
: svn checkout <nowiki> https://white.stanford.edu/ee/iset-4.0 </nowiki> <br>
+
-
* If you just want a portion of the very large repository, you can use
+
-
: svn checkout <nowiki> https://white.stanford.edu/ee/pdcprojects/L3pipeline </nowiki> <br>
+
-
: svn checkout <nowiki> https://white.stanford.edu/ee/pdcprojects/LEDms </nowiki> <br>
+
-
 
+
-
== Checkout: Windows ==
+
-
[[Image:svnCheckOut.jpg|thumb|300px|PDC Checkout]]
+
-
 
+
-
We suggest using[http://tortoisesvn.net TortoiseSVN] - ''you'll need to restart after the install''. You can [http://tortoisesvn.net/downloads download TortoiseSVN from here].
+
-
 
+
-
After the installation, check out the pdcprojects directory by
+
-
 
+
-
* Right click for the context menu
+
-
* Select the SVN Checkout option
+
-
* This will bring up a window that looks like the one at the right
+
-
 
+
-
You should fill in the relevant repository (or just the specific directory within the respository).  You can specify where you want the folder or repository to be located on your local machine.
+
-
 
+
-
There are many TortoiseSVN tools that make it easy to resolve conflicts and check for updates and commits.
+
-
 
+
-
== Repository Browsing ==
+
-
 
+
-
To have an overview of the main PDC and ISET repositories, you can use the browsing feature of the SVN repositories.
+
* [http://white.stanford.edu/trac/pdcprojects/browser PDCProjects]
* [http://white.stanford.edu/trac/pdcprojects/browser PDCProjects]
* [http://white.stanford.edu/trac/pdcsoft/browser/trunk PDCSoft]
* [http://white.stanford.edu/trac/pdcsoft/browser/trunk PDCSoft]
* [http://white.stanford.edu/trac/iset-4.0/browser ISET-4.0]
* [http://white.stanford.edu/trac/iset-4.0/browser ISET-4.0]
-
 
-
These links show you the repository directories as well as the history of the different files and directories in the repositories.
 

Latest revision as of 08:45, 23 April 2016

The software we develop for this work is distributed through our github repositories.

Here is a local page describing the basic GIT operations.

Contents

[edit] Software

We are part of Stanford's Center for Image Systems Engineering. Several projects related to image systems engineering are stored at these accounts.


The github repositories maintain a series of programs related to our database and computational work in our project on scientific transparency.

See also:

[edit] Data

We are storing critical data sets for

on the Archiva server. We interact with these databases using the RemoteDataToolbox.


[edit] Running Matlab Remotely

We use Microsoft Remote Desktop to login to servers and run remotely.

[edit] Virtual Box

Andy Lin has us running Ubuntu on Windows machines using VirtualBox. This is a free virtual machine system from Oracle.

Install Virtual Box on a Windows machine. Then, download a version of Ubuntu.

You can then create a virtual instance of Ubuntu that will run on the Windows machine. Some suggestions:

  1. Set up the Machine properties for more than 1 processor.
  2. Set an execution cap that is less than 100 percent so the Windows side of your machine can continue to run
  3. Use more RAM (e.g., 4 or 5 GB).
  4. Use a virtual disk of 15-20GB at least.

We will try to make the notes here a little better as more people go through the process. Though it may be that only Andy and Brian have Windows machines.

[edit] Writing Matlab Files

When you are writing Matlab scripts or functions, please follow some guidelines in order to allow others to be able to use and understand your code and its outputs. It is important that the code is written in a readable modular fashion so that other people in the lab (now and in the future) may benefit from and work with your code.

At the top of each of your files, please add a header describing what it does. The suggested format for a simple example function appears below:

 function ycbcr = myrgb2ycbcr(rgb)
 % Converts an image using the color conversion used in JPEG
 % 
 %     ycbcr=myrgb2ycbcr(rgb)
 %
 % A few sentences about the function there ...
 %
 % ...
 % 
 %
 %INPUT:
 %   rgb:        Double array giving an RGB image with values between 0 and 255
 %               size(rgb)=m x n x 3
 %
 %OUTPUT:
 %   ycbcr:      Double array giving a YCbCr image with values between 0 and 255
 %               size(ycbcr)=m x n x 3
 %
 % See also:  Related functions
 %
 % Examples:
 %    Examples go here
 %
 %
 % (c) Copyright notice


Additional comments or notes can be added after the description of the output(s).

[edit] SVN (DEPRECATED)

[edit] Repository Browsing

As far as I know, everything is now on github or the RDT (Maven/Archiva) repository.

I leave these links here just in case we forgot something.

Personal tools