Homework 3: Genre Classification

Due at 9:30am on July 27

With this lab, you will explore text classification with data from a variety of speech and text sources. The task is to classify which of the 7 genres a document is associated with:

• speech:
  1. broadcast news (BN);
  2. talkshows, referred to as broadcast conversations (BC);
  3. conversational telephone speech (CTS)
• text:
  1. newswire (NW),
  2. weblogs (Weblogs),
  3. amazon book reviews (AmazonBooks),
  4. Amazon appliance reviews (AmazonAppliances)

The task is to classify each test document as one of these genres. You can download the data from link. Training and test partitions are specified. There is no dev set, so if your explore different options then you will need to evaluate alternatives by holding out part of the training set, or doing cross validation on the training set. Do not run multiple configurations on the test set. The data is organized in directories by genre, so that provides your answer key.

You may explore any approach for classification that you want, but a simple method is to use a Naive Bayes classifier with word counts as features. There are several options available in the RAINBOW text classification toolkit:
http://www.cs.cmu.edu/~mccallum/bow/rainbow/,
which is easy to use, but there are other toolkits that you are welcome to try. Some options include:

• svmlight ( http://svmlight.joachims.org/ ),
• Maxent (http://www.cs.princeton.edu/~schapire/maxent/
• the MALLET toolkit, including several classification algorithms http://mallet.cs.umass.edu/
• the icsiboost ( http://code.google.com/p/icsiboost/, an open-source implementation of "Boostexter," which is an Adaboost classifier specialized for text problems.

A modified version of the rainbow source code is available here http://ssli.ee.washington.edu/people/brianhutchinson/src/bow-20020213-patched.tar.gz if you want to use RAINBOW and are having trouble compiling with a recent version of gcc.

The simplest approach would be to use word counts as features, but you may want to try POS tags. A tagger is available at: http://nlp.stanford.edu/software/tagger.shtml

Using punctuation as "words" may be useful. The documents are truncated in length, so don't use document length as a feature.

What to turn in:

• One page write-up discussing the things you tried and why, how you did tuning (held out data vs. cross-validation and associated details) if at all, and the results for the best-case model on the test set. Provide URLs for any resources that you used other than those listed above.

Note from Jinho on POS-tagged data

For those who are not so familiar with NLP tools, I sentence segmented, word tokenized, and part-of-speech tagged the data. You can download the tagged data from the following link.

http://www-nlp.stanford.edu/danj/genre-tok.tgz
http://www-nlp.stanford.edu/danj/genre-pos.tgz

'genre-tok.tgz' contains non-tagged tokenized data and 'genre-pos.tgz' contains pos-tagged data. Each file in 'genre-pos.tgz' follows the following format.

FILE ::= (SENTENCE)+
SENTENCE ::= TOKEN(<SP>TOKEN)*
TOKEN ::= FORM_POS
<SP> ::= space
FORM ::= word-form (including punctuation)
POS ::= part-of-speech tag of the word

You can find the list of part-of-speech tags from the following link: http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html