ISET-Depth-Old

From VISTA LAB WIKI

Jump to: navigation, search

Contents

[edit] Overview

The method relies heavily on Dave Brainard's Rendertoolbox (RTB). RTB is built on top of two other packages: Maya and Radiance.

Maya is an object modeling package that is freely available to universities (see below). This is a means for creating 3D objects and placing them in a scene.

Rendertoolbox reads the Maya objects and allows the user to specify spectral reflectance functions for the objects, the illuminant, and to specify bi-directional reflectance distribution functions (BRDF) for the objects (we haven't tried the last part).

Rendertoolbox combines the Maya data and these spectral functions in a wavelength-by-wavelength calculation and passes them to the Radiance package. That package creates the rendered image in each waveband to produce a spectral output.

ISET combines this output into an ISET scene structure that contains information about the scene spectral radiance, illuminant and depth map. These 3D scenes can be analyzed by the ISET optics, sensor and other modules to characterize depth-of-field and defocus, as well as other properties.

For another overview see http://white.stanford.edu/teach/index.php/Andy_Lin

For the user manual and technical report, please see http://www.stanford.edu/~ydna/Scene3DManual.pdf

[edit] Computer Requirements

We assume you are on a Linux system. We don't have installation instructions for other stuff.

We need 32-bit Matlab (there is no 64 bit version of XXX).

We have installed on Red Hat's Fedora OS version XXXX

We have also installed in Ubuntu 10.10 (Maverick).

Not sure about graphics cards requirements - probably none.

[edit] Download and Installation

We list the Ubuntu scene3D installation instructions separately from the Red Hat scene3D installation.

We are just starting to work on the Windows scene3D installation.

[edit] RenderToolBox

RenderToolBox is a collection of rendering tools developed by Professor David Brainard from University of Pennsylvania. RenderToolBox allows for the rendering of 3D multi-spectral scenes, using various rendering engines. RenderToolBox was modified slightly to allow for the output of depth-maps. Please refer to the RenderToolBox website for more details: http://rendertoolbox.org/

[edit] Initializing RenderToolBox

  • Start Matlab
  • Your path should include ISET, RenderToolbox, SimToolbox, psychtoolbox-3. Also, it is best to remove the SVN directories from your path. An example of a fairly complex startup program from wandell is:

PDC Startup Example

A program to remove .svn directories from your path is here

SVN Remove Path

Now, you should be ready to follow the example below.

[edit] Simple RenderToolBox Example

Change into the scene3D directory.

This is a simple example that loads a scene of a sphere on a table, in a room, along with its depth-map into an ISET scene. Run this example as a simple verification if the installation was successful or not. This part is designed to run on a Linux box that has all of the tools installed.

In Matlab, cd into the RenderToolbox directory and then into the sub-directory examples/table_sphere/

chdir(fullfile('/usr','share','RenderToolbox','examples','simple_square'))


Note that this directory is the one that will contain your scene. Each scene will have its own root directory. In this example, the example scene is called table_sphere, so it has it's own table_sphere directory. All the necessary files to render this scene will be in this directory, which we will refer to as "scene root".

Run the following series of commands from inside the Matlab prompt

unix('./clean.sh')
Render_BatchRender

On ochre this took about 10 sec.

To verify the output, change into sub-directory within the table_sphere directory, monitor_image_data, and open a .jpg file for verification.

im = imread('monitor_image_data/rad_MonitorImage.jpg'); image(im)

The multi-spectral data will be in the image_data directory. ISET will then use this data to construct multi-spectral scenes.

d = load('image_data/picMat_rad.mat'); 
vcNewGraphWin; imagesc(d.picMat{10}); colormap(gray)

[edit] Setting up Maya objects for Rendertoolbox

This section describes how to take Maya objects and render them as a set of multispectral files that will be converted into an ISET scene.

Either create a scene of Maya objects, or download Maya objects from an example site and place them in a Maya scene.

(At this point, we have a few scenes (table_sphere) that contain a room and wall and so forth. We have only learned how to place new objects inside this scene. We don't know yet how to arrange the geometry of a full scene from scratch.)

  • Open the scene using Maya (We use Maya for Windows).
    • If you have downloaded an object to place in a scene, you need to use Maya to scale the object and place it at the correct general location of the scene/camera. Some Maya objects are actually composed of multiple objects - you must scale, shift, and export all of them.
    • In Windows->Settings/Preferences->Plug-in Manager, make sure objExport.mll is loaded (you can select auto-load if you will be using it frequently).
    • Select object (must select all objects if your object consists of many objects)
    • File...Export Selection... Name the exported file the same as the Maya objectName field. Here is more information on how to export from Maya.
  • IMPORTANT: The exported objects need to be edited before RTB handles them properly. In Linux, using, say, gedit or xemacs, open and edit the .obj file. (It is a text file). Replace the strings 'usemtl XXX:YYY' with 'usemtl objectName'. Thus, if your object is 'Bench' and there is a line such as 'usemtl Bench1:default1' replace it with 'usemtl Bench'.

[edit] Using Rendertoolbox to create a multispectral image from Maya objects

The RTB builds multispectral images from data and parameter files stored in a scene directory. A relatively easy to way to begin creating an image of a scene is to modify an existing example. You might begin by copying the files in the "table_sphere" directory in "$RenderToolbox/examples/" to a new directory where you will do your work.

  • Replace the objects in the scene_objects directory with the new Maya objects. In an example we did, we copied the table_sphere directory and into banana_bench. We then placed a set of maya objects called Ball_Thing1, banana1, and some others into the scene_objects directory. We removed the old objects.
  • Edit the parameter files.
    • Edit the objectProperties.txt in the scene root directory by adding the name of each object and setting the fields. There are a number of fields that must be filled in to this tab-delimited text file. To get started you can copy and paste previous entry if don't know what to put here.
    • Edit the conditions.txt file in the scene root directory. Mainly you can set up the scene spatial resolution (300 is OK for testing), set up the wavelength sampling (400 100 3 produces samples at 400, 500, 600 which is good for testing). It is nice to have a good scene name, as well.
  • Run Render_BatchRenderClean, a RenderToolbox script that cleans the directory and then invokes radiance and rpict. If all goes well, the multispectral data will be created and a simple RGB image of the result will be placed in the monitor_image_data directory.

[edit] Modifying RenderToolbox parameter files

  • Add the name of new objects to objectProperties.txt (which is in the scene root folder). The objectProperties.txt contains the texture/reflectance information for each object. Refer to the rendertoolbox wiki for details.

Similarly,

  • To add/edit lighting, edit lightProperties.txt
  • To add/edit rendering parameters, edit conditions.txt
  • To change view points, edit ./view_files/view.vf (or the current view point file specified by conditions.txt)

Refer to the rendertoolbox wiki for details.

When the directory is setup properly, within Matlab you can

  • Change into the scene directory
  • Run Render_BatchRender
  • Or, if you are running multiple times, use Render_BatchRenderClean.

[edit] Importing RTB multispectral data to an ISET scene structure

Scene3D is a component of ISET which interfaces with RenderToolBox, and allows for the RenderToolBox output to become an ISET scene, with a depth map. Scene3D also allows for depth-of-field simulation of multispectral scenes with provided depth-map.

[edit] An example

To load the RenderToolBox multispectral output and a depth map into an ISET scene structure, first change directory into a scene directory, then run:

  • scene = s3dRT2Scene([pwd '\image_data\picMac_orange_rad.mat'], 400:10:700);

The depthmap is stored in scene.depthMap.

The following command is under development:

Finally, to render the scene with optics blurring, run:

  • s3dBlurScene

This command loads the scene from table_sphere into an ISET scene, then apply default blurring properties to the optical image. The command uses defaults, but you can run the code with different optics parameters set. We are still in the process of debugging and validating the blurring code.

Personal tools