ISET-Depth

From VISTA LAB WIKI

Jump to: navigation, search

We are combining code from the RenderToolbox, SimToolbox, Radiance, ISET, and Maya to build 3D models with full spectral radiance data. We process these models through ISET to develop a more general rendering with depth of field and appropriate blurring.

The programs specific to this project are in the pdcprojects repository in the scene3d directory.

Andy Lin, Joyce Farrell and Brian Wandell

Contents

[edit] Overview

ISET can simulate the responses to 3D scene data; these are spectral radiance data that include information about the distance from the lens to each point in the scene.This page describes how to create synthetic 3D ISET scenes.

See ISET-Depth-Old for the previous configuration of Scene3D using Rendertoolbox. We have since moved away from this toolbox in favor of our custom configuration of a heavily modified version of PBRT.

[edit] PBRT Multispectral Modification

The next phase of the Scene3D project will consist of modifying PBRT to allow for multispectral inputs and outputs. The reason why PBRT is chosen is because PBRT has some nice native support for texture mapping, material modeling, is modern, and is easy to modify. There is already some native PBRT support for spectral imaging. However, the biggest concern is that it does not allow for spectral outputs.

Also, the multispectral inputs are somewhat limited. For example, PBRT does not natively allow for multispectral image maps for textures. This next phase of the project will consist of modifying PBRT for multispectral input and output.

Here are instructions on how to install our modified version of PBRT. Note that this has been tested on 64-bit Ubuntu version 12 systems only.

  • Download libtiff [[1]], ilmbase [[2]], openexr [[3]]
  • Install libtiff
unzip tiff-3.8.2.zip
cd tiff-3.8.2.zip
./configure; make; sudo make install
cd ..
  • Install ilmbase
unzip ilmbase-1.0.2.zip 
cd ilmbase-1.0.2
./configure; make; sudo make install
cd ..
  • If openexr complains about something about ilmbase, install libilmbase6 and libilmbase6-dev using Synaptics Package Manager.
  • Install openexr
unzip openexr-1.7.0.zip
cd openexr-1.7.0
./configure; make; sudo make install
cd ..
  • Note: you may need to run "sudo ldconfig" first if you are having trouble installing openexr.
  • For some systems, before installing openexr, you need to install zlib.
    • Download zlib 1.2.7 from [[4]]
    • Install zlib using the following commands:
tar -xf zlib-1.2.7.tar.gz
cd zlib-1.2.7
./configure
make
sudo make install
  • Depending on your configuration, there may be issues installing openexr. If there is an error while compiling openexr that says 'memcpy' was not declared in this scope, use the following fix:
    • Add '#include <cstring>' to the top of /usr/local/include/OpenEXR/lmathMatrix.h' and './exrenvmap/blurImage.cpp'
  • An additional dependency for Scene3D spectral PBRT is the gsl library. Download version 1.14 here: [[5]]. Install gsl-1.14 by running the following commands
tar -xf gsl-1.14.tar.gz
cd gsl-1.14
./configure
make
sudo make install
  • Another dependency for PBRT is Bison. Install Bison using the Ubuntu Synaptic Package Manager.
  • Another dependency for PBRT is Flex. Install Flex using the Ubuntu Synaptic Package Manager.
  • Download the Scene3D version of PBRT using the following link via git: [[6]]
    • Unzip these files in any location.
    • Change into the src/ directory
    • Run the following commands to command spectral PBRT
make clean
make
  • Install ISET. Here is a website describing ISET, including information on how to obtain it: [[7]].
  • Download the Scene3D Matlab files here: [[8]]
    • Unzip the files to any location.
    • Edit 'pbrtHome.m' to reflect the installed location of the Scene3D version of PBRT.

[edit] PBRT Modification Details

See PBRTOutput for details on how PBRT was modified to allow for multispectral output.

See PBRTConversionMatrix for details on how PBRT was modified to allow for a conversion matrix multiplication, after processing.

See PBRTFileFormat for the input file format for conversion matrices.

[edit] Random Notes

  • some pbrt scenes use bump-mapping - basically it uses a height-map, and uses the height-map to compute normal vectors, to approximate a bumpy surface. In the end, this is only an approximation, and I wonder if we can modify pbrt to utilize more advanced techniques than bump mapping.
  • pbrt can measure subsurface scattering - see chapter 16 - some of this stuff is quite advanced
  • in teapot subsurface scattering example,
    • sigma_prime: Reduced scattering coefficient of the volume, measured in mmˆ-1
    • sigma a: Absorption coefficient of the volume, measured in mmˆ-1.
    • what do these parameters mean?
  • “kdsubsurface” material is easier to specify the "look"
    • Kd: diffuse reflection term
    • Kr: specular reflection term
    • meanfreepath: average distance light travels before being scattered
  • working on extracting depth map - this has been finished!
    • in film.cpp... we need to modify AddSample()
    • however, can't find the original call to Addsample()... this is very curious
    • may need to change the Ray data structure to allow for storage of depth (in geometry.h)
    • add a Ray & currentRay input parameters to AddSample() in image.cpp, needed to change image.h and film.h as well
    • added Z data member to Pixel struct in image.h
    • 3/4 - finished getting depth map and importing into ISET, however, there are PROBLEMS with compatibility with ISET + drawing depth maps etc. - should ask Prof Wandell how exactly this works.
  • working on blender to pbrt converter
    • find files in this directory: "C:\Users\Andy\AppData\Roaming\Blender Foundation\Blender\.blender\scripts\bpydata"
  • environment maps are in lat/longitude coordinates. in other words, they are in spherical, with theta and phi mapped to u and v.

[edit] PBRT Realistic Lens

  • The realistic lens support, as specified by Prof. Pat Hanrahan's Raytracing class, CS348b, has been implemented
  • Here is a link to the 2012 CS348b class website [9]
  • Here is a link that describes how the Realistic Camera class was specified [10]
  • To summarize what the changes are, a new "Realistic Camera" class was added that allowed for ray tracing through multiple spherical lens elements. In order to simulate this, a vector form of Snell's law was utilized. The rays originate from the sensor and are uniformly shot at the back lens element to allow for an unbiased rendering.

[edit] PBRT Lightfields

  • We have devised an experimental setup, where the light field is captured between the back element of the realistic lens and the sensor.
  • We modified PBRT to keep track of intersections with the (u,v) and (s,t) planes. These intersection locations are recorded, along with the radiance from the ray.
  • In order to render an image from a light-field, we simply project rays at the back lens element, as if we were doing the regular ray-tracing pipeline. However, instead of actually ray-tracing through the lens elements, we use table look-up from the existing light-field, by first calculating the intersection of that ray and the (u,v) and (s,t) planes and using table look-up.
  • See the class report for more details and the source code. [11]
  • modified files: realistic.cpp/.h; lightfield.cpp/.h (added); samplerrenderer.cpp
  • Special thanks to my project partner Nicolas Hippenmeyer for making PBRT lightfields possible.

[edit] TODO and Miscellaneous comments

rview (Radiance program that allows for interactive 3D viewing) is currently broken and should be debugged.

scene3DInitialize.m needs to be added to SVN

very early scene3D files should be packaged and sent

some preview visualization technique for scenes would be great


  • add flexible input range - this is a challenge... there are many vectors that are initialized with the currently provided constant. All of them would need to be modified to allow for dynamic allocation... plus, is it any faster? 31 channels vs. 310? need to try this out... see if it makes a difference!
  • add flexible output (specify output by a matrix perhaps) - this is done. may need to polish
  • add Matlab interface so ISET can read the files! - this is done, but need to improve this interface.
  • get the depth-map - look at the pbrt website and look at Extended channel film plugin for reference.[12]
  • Obtain multispectral textures/light maps/materials/measured subsurface scattering
    • obtain bump-maps for material realism
  • cleanup the DOF rendering algorithm that takes into account partial occlusion -> package this, perhaps into ISET
  • allow for multispectral textures
    • what is the best file format to use for this?
  • allow for multispectral light maps (environment map)
  • make some nice example scenes (buildings, outdoors?)


Some notes

  • pbrt uses bison, a gnu parser?
  • parsing is taken care of in the lex and ycc files, core/pbrtlex.ll and core/pbrtparse.yy
  • Important to figure out how to input NEW PARAMETERS
    • paramset.cpp/ paramsset.h has some answers
    • pbrtparse.cpp/pbrtparse.yy has the bulk of the parsing procedures
    • api.c/h ...
    • InitParamSet in pbrtparse.yy is a good example of using paramset functions! don't need to worry about nItems because it detects that for you. We might need to make a new PARAM_TYPE enum for the transformation matrix! Need to make a new keyword for it too. Initially can just read from the .pbrt file, eventually want to have it read from a text file... although this will make it harder to implement...
    • Api.cpp -> Film *MakeFilm .... perhaps this is where we put the new transformation matrix!!!!!
    • perhaps use ParamSet::AddSampledSpectrumFiles as inspiration on the input interface
    • add an entry to pbrtparse.yy InitParamSet for transformation matrix
    • proposed data flow.... pbrtparse.yy/cpp -> initialize param set (Create new entry for matrix file?)... -> api.cpp MakeFilm .... -> Image.cpp, CreateImageFilm .... extract matrix filename from param set and perform parsing... may need to use paramset class to perform actual parsing (ParamSet::AddSampledSpectrumFiles inspiration)
    • .pbrt files require you declare the type before specifying the parameter type (use string for the filename of the matrix)


3DS Format to PBRT

  • meshes export ok
  • however, materials are a mess - need to go into 3ds and adjust all the materials to PBRT types -> is there an easier way to go from 3ds to pbrt?!!
  • need to specify a camera position, or else will give an error


Blender to PBRT

  • must use Blender version 2.48a!!
  • put .py file in the directory that all the scripts are in (use the search function to find this)
  • go to Render - then you will find PBRT export selection in there
  • by default, there will be an infinite light source -this lighting is very flat
  • do lights specified in blender work in the converter?! not sure about this
  • C:\Users\Andy\AppData\Roaming\Blender Foundation\Blender\.blender\scripts\bpydata is the location of the exported pbrt files
  • lights don't work! - lights DO work - the infinite light just drowns them out!! - remove infinite light perhaps, or make much dimmer
  • lights in PBRT are using the global coordinate system!! (not eye coordinates/clip coordinates)
  • must use .tga or .exr texture maps! but these WILL work!
Personal tools