NIMS Contract


Jump to: navigation, search


[edit] Scope

The objective of this short project is to continue work on NIMS to produce a system that can be used to securely upload, store and access neuroimaging data. This short project should provide an attractive demonstration with substantial completed functionality. This should demonstrate

  • how NIMS supports basic image and scan activity management.
  • how easy NIMS is to use

Dr. Jorge Phillips will be responsible for implementing the deliverables below.

Anand Iyer will also be working on this project from the Stanford side with a general availability of between 15 and 20 hrs per week over the next months.

[edit] Project Management

The deliverables below will be carried out within the time-frame of this agreement on a best-effort basis.

Jorge will provide project technical direction and management of Anand's work on NIMS.

Bob will support the local management of Anand and the computer systems.

[edit] Deliverables

[edit] Base code


Base code is loaded in Stanford servers.


Ensure base code is operational on the target server (

[edit] NIMS Request Processor

[edit] Architect and design a persistent backend request processor

Current: NIMS implements a simple message passing interface to a backend service called MQ. MQ is brittle, requests sent to the backend are not persisted, so may be lost if either client or server dies, and deadlocks may happen due to no resynchronizing capabilities in MQ

Planned: We need to architect and design a persistent request processor that works a layer above TCP/IP as opposed to MQ.

[edit] Implement NRP, the NIMS request processor

Current: No code yet

Planned: Implement an extensible, scalable distributed backend request processor for NIMS that a) allows several types of backend computational requests; b) provides request persistency; c) is self-repairing to an extent, in the sense that both client and server should be able to recover from failures during request processing.

[edit] Updated exam upload, neuroimaging object (NIO) indexing, and content download

[edit] Modified exam upload structure


The current system provides a flexible exam content structure and images from a series are uniquely identified by their series and acquisition number. There is no naming convention for exam content upload (i.e. NIMS does not interpret series directory names as having any meaning e.g. Anatomy, Pfile and DTI content).


We will allow uninterpreted content to be uploaded and associated to an exam or its associated series. The new functionality will also allow series to be grouped into a hierarchical file structure, such as:

  • An exam to be uploaded is a compressed directory of trees.
  • At upload time, an Exam entity is created in the database from which all uploaded content will be referenced.
  • Each subdirectory in an exam tree(including the root directory) contains a number of image files (DICOM slices), P* files, and uninterpreted content such as text files, PDFs, functional annotations, etc. to be indexed into the database.
  • Subdirectory image files are grouped into series objects according to <series#, acquisition#> or according to P* conventions. At upload time all series objects in each subdirectory will have their descriptors loaded into the NIMS database as Series entities and their raw image data converted to separate .nii image volumes
  • Uninterpreted content in the root directory is associated at upload time to the exam object being created
  • Uninterpreted content in a non-root directory that has only one series is associated to that series
  • Uninterpreted content in a non-root directory that has several series is associated --arbitrarily, TBD -- to the series in the subdirectory with the earliest acquisition time. If a user wants to embed several series in the same directory, each with separate uninterpreted content, then the user must place each such series into a separate single-series sub-directory and the uninterpreted content placed there.
  • Directory names in the upload file will not be discarded. They will be used as the default description for the NIO.

[edit] Pipelined Exam Uploading


When a researcher is at the scanner in the Lucas Center, uploading an exam includes completion of header analysis of all series and validation of header information. For a normal scan session with anatomy, functional, screensaver and DTI series, this may involve opening, reading and analyzing thousands of files, which can be a slow process during which the researcher cannot close the upload session, work on other tasks in NIMS, or leave.


We will design a new pipeline stage, similar to the current one for exam and series header validation, which will upload an exam to the NIMS staging area, with minimal analysis and a partial exam header. The user will be given the opportunity of verifying basic exam header data and then proceed with the exam content upload. No interpretation of exam content will be done. The delay experienced at the scanner station will be limited to that involved in verifying minimal information in the exam header (e.g. exam ID and perhaps subject) plus the actual data upload. An asynchronous process will be started to read the uploaded data, create exam and series headers, which will trigger at completion the existing validation, archiving and structure creation stage.

[edit] Allow a user to add uninterpreted content to an existing exam


NIMS currently allows uploading a single series into an exam, with no uninterpreted content.


We will extend this to allow a directory with only uninterpreted content to be uploaded to any existing exam.

[edit] Exam and Series content upload, download and bundling


NIMS currently allows users only to download individual NIFTI files.


  • Provide functionality to upload/download uninterpreted content of different types into exams and series. The user should be able to specify the type of content being uploaded, and provide short (compulsory) and long (optional) descriptions associated to the content object. Content uploaded in this manner is stored as-is in NIMS.
  • Provide functionality to download an exam bundle, i.e. a single zip file containing all series volumes and associated content for an exam
  • Provide similar functionality to download a series bundle, i.e. a single zip file containing the series volume and its associated content.
  • Provide functionality to view all content associated to an exam or to a series in a searchable, sortable by column tabular form.
  • Implement functionality so that authorized users (initially root) can define and alter the types of uninterpreted content in NIMS and the order they appear in pulldown menus in the user interface.

[edit] Content Upload/Download Improvements


  • Improve the content download process so that all NIFTI images that are downloaded have a unique NIMS id string inserted into the header description field. This identifier will contain the unique NIMS database ID for the container NIO as well as a date code that defines when it was downloaded. We think that any time a file is touched, though, it should get a new ID.
  • Downloaded volumes will be anonymized, with no HIPAA data in the NIFTI headers
  • Implement a system where the user can select specific sub-directories to be assembled into a zip file and downloaded.

TO DISCUSS: The more structure we specify at exam upload time, the more efficiently we can tag data attributes for download (e.g. ANATOMY) and classify them accordingly in the exam structure so that at download time. The researcher can obtain a more meaningful directory tree structure at download time:

  • [subject code][date string] (e.g., "jp100219")
    • Raw
      • Anatomy
        • inplane.nii.gz
        • ss.nii.gz
        • t1.nii.gz
        • dti
          • dti.nii.gz
      • Pfiles
        • [all pfiles and efiles for this exam; use the filename as uploaded]
      • Notes
        • [any additional files]

[edit] API and GUI for NIO Associations (Derivations)

[edit] Implement and document the back-end API for deriving new NIOs from existing NIOs.


There is no functionality in NIMS to derive new content from existing content.


  • Define a clear method for modifying the database schema
  • Define a clear method for for modifying the object-relational maps as required to access new structures
  • Implement and document a derivation mechanism API

[edit] Implement and document two specific GUIs to allow users to create new NIOs from an existing NIO, such as a series volume.


3.4.a The user can download an anatomical image. After segmenting this image, they will be able to upload the resulting tissue classification nifti image as a new NIO that is associated with the original anatomical image. In a later phase, when NIMS id strings are implemented in volumes, if the uploaded image contains a NIMS id string in the header, then the associated anatomical image can be automatically found. In this phase, the user will be able to select the parent NIO from the database and perform the upload.

3.4.b The user can download an anatomical or functional image. After defining an ROI, they can upload the ROI object (either as a Matlab (.mat) file or a nifti image) into NIMS. This file will be associated (tagged) with the original NIO. ROIs may need to be associated with multiple NIOs. For example, a functionally defined ROI will have a functional dataset and it may also have an anatomical NIO and a tissue classification NIO. In a later phase, when NIMS id strings are implemented in volumes, if the uploaded image contains a NIMS id string in the header, then the associated NIO can be automatically found. In this phase, the user will be able to select the parent NIO from the database as in 4.b.1 and upload the corresponding ROI.

[edit] Scanning Activity Management Reporting


There is no reporting in the current version of NIMS


Design and implement reporting functionality for generating scanning activity reports. One or two such reports will be implemented (within time constraints). The objective here is not to produce a full blown reporting engine, but rather to implement sufficiently complete machinery and documentation, such that new reports, etc. can be added easily into NIMS. This will require

  • Modification of the data model to create a scanning activity table for all exams uploaded into NIMS. This table will join grants, experiments, exams, experimenters, and subjects with scan time.
  • Implement simple queries and tabular displays to extract report data from the scanning activity table between two dates based on specific values for one or several columns. Examples of this functionality may be e.g. to extract the scanning activity between two dates for grant NIH-2010-001, which will generate a table with experiments, exams, experimenters and subjects filled out for this grant, and a total scan time for the grant. Another example would be to report the total scan time for experimenter Joe on experiment X between two dates. This should produce a time in hours and minutes, for example. Even another example may be to report all scans done for experiment Y between two dates. In this case the experimenter, subject and exam fields would be shown, as well as a total exam time.
  • We will implement a report generation GUI that produces and aggregates the corresponding results as described above.

[edit] Consent, Assent and Payment Forms


There is no facility for associating legal forms signed by a subject participating in an experiment, to the subject's information in the NIMS database.


Implement a facility for associating uploaded subject consent, assent, and payment forms (usually scanned PDFs or JPEGimages) to a subject in NIMS. These documents will also be associated with an experiment. E.g.:

  • Subject Joe Smith might have the following documents:
    • consent for Color Experiment
    • consent for Reading Experiment
    • MR screening form for January 3, 2009
    • MR screening form for March 21, 2009
    • payment form for Color Experiment
    • payment form for Reading Experiment

[edit] Improved Login Functionality


Login is not robust enough, need to improve login security to conform to the Security Audit document. Currently upon detecting a credentials problem the wsgi application enters into a 15 sec delay during which neither the interface is refreshed nor an error message is put up. While this is ok for a bot that is attacking the system, this behavior is not acceptable to actual NIMS users that mistyped their password or login name.


Currently the only built-in deterrent is a delay after an invalid login, which blocks the wsgi application. Upon such a failure control should return to the interface immediately, an error flagged, and a refractory period started.

The following are the objectives of this task:

1. Implement the delay after invalid login functionality in a way in which the login interface becomes refractory to logins for 10 sec without blocking the server thread, and after putting up an error message back to the user interface.

2. Implement functionality so that NIMS will allow only three login failures before blocking logins for an account regardless of the time interval between failures.

3. Collect and store in each user's login record in the NIMS database, information about good and bad logins including IP from which they were attempted, as well as date and time information. This is needed to make NIMS consistent with the security audit document.

4. Implement functionality to handle the case when a failed login is not due to an erroneous password, but to having provided a non-existent user name for login. Such a case should be logged so that sysadmins can track the originating IP and if needed block access to NIMS from that IP.

[edit] Documentation


NIMS currently includes a substantial amount of documentation, both at the code level and as documents in the source code tree. The system has very scant user documentation at this point.


In this task we will complete the application documentation for the new functionality being added, and we will provide more user documentation.

1. Complete documentation of the core NIMS application programming interfaces in the NIMS code tree, so that it is version controlled with SVN. The NIMS wiki can later link to the relevant documentation files via the TRAC SVN code browser.

2. Complete NIMS tutorial user guide documentation (as part of the NIMS wiki).

[edit] Security Audit Document


Carry out a security assessment of the current implementation and prepare an security assessment document for HIPAA that explains in detail the security mechanisms currently in place in NIMS as well as all necessary measures that are being taken to safeguard human subject data. There are two objectives of this document: first, to enable you to guarantee to other potential users that the architecture and implementation of the system satisfies all required security and information protection standards, and second, to identify security areas of improvement to tackle, that depending on time and resources may or not be carried out as part of this project.

HIPAA summary of Protected Health information (PHI), from HIPAA site:

The Privacy Rule protects all "individually identifiable health information" held or transmitted by a covered entity or its business associate, in any form or media, whether electronic, paper, or oral. The Privacy Rule calls this information "protected health information (PHI)."

“Individually identifiable health information” is information, including demographic data, that relates to:

   * the individual’s past, present or future physical or mental health or condition,
   * the provision of health care to the individual, or
   * the past, present, or future payment for the provision of health care to the individual,

and that identifies the individual or for which there is a reasonable basis to believe it can be used to identify the individual. Individually identifiable health information includes many common identifiers (e.g., name, address, birth date, Social Security Number).

The Privacy Rule excludes from protected health information employment records that a covered entity maintains in its capacity as an employer and education and certain other records subject to, or defined in, the Family Educational Rights and Privacy Act, 20 U.S.C. §1232g.

De-Identified Health Information. There are no restrictions on the use or disclosure of de-identified health information.14 De-identified health information neither identifies nor provides a reasonable basis to identify an individual. There are two ways to de-identify information; either: (1) a formal determination by a qualified statistician; or (2) the removal of specified identifiers of the individual and of the individual’s relatives, household members, and employers is required, and is adequate only if the covered entity has no actual knowledge that the remaining information could be used to identify the individual.

Notes: cover database and web services security, executive summary, security procedures

[edit] Schedule of Deliverables

Wave 1:

  • Task groups 3.1, 3.2.1, 3.2.2, 3.3.2, 3.3.3, 3.3.4, 3.7, 3.9 and start task 3.8

Wave 2:

  • Task groups 3.4, 3.5, 3.6, and complete task 3.8 throughout

Future work:

  • Task groups 3.1, 3.5

[edit] Milestones

Task 1 has already been completed.

  • Milestone 1: 3.1, 3.9, 3.2.1, 3.2.2
  • Milestone 2: 3.3.2, 3.3.3, 3.3.4, 3.7
  • Milestone 3: 3.4
  • Milestone 4: 3.5
  • Milestone 5: 3.6

We have replaced 3.3.1. by 3.2.1 and 3.2.2 to implement NRP, which was not part of the original set of tasks. 3.3.1 will be part of a later phase of the project.

We have replaced the original 3.3.4 by

A) a new task 3.3.4 to implement bundled downloading and content type management and a new task 3.7 to implement secure login functionality and

B) a new 3.3.5 task corresponding to the original 3.3.4, which will be part of a later phase of the project.

[edit] Status

  • Task 3.1: Done
  • Task 3.2.1: Done
  • Task 3.2.2: Done
  • Task 3.3.2: Done
  • Task 3.3.3: Done
  • Task 3.3.4: Done
  • Task 3.4.1: Done
  • Task 3.4.2 Done
  • Task 3.5: Done (UPDATED) Jorge 15:19, 30 September 2010 (PDT)
  • Task 3.6: Done
  • Task 3.7: Done
  • Task 3.8: Ongoing. Completing internal documentation page. (UPDATED) Jorge 15:19, 30 September 2010 (PDT)
  • Task 3.9: Done
Personal tools